2024-09-17T07:36:35.032 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T07:36:35.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T07:36:35.133 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_latest} thrashers/cache thrashosds-health workloads/rbd_nbd} email: sangadi@redhat.com first_in_suite: false job_id: '7908725' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - mons down - mon down - out of quorum sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: enable_coprs: - ceph/el9 extra_packages: - rbd-nbd flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbxAaFhqneXjYqjKjSehs56lFBY4Uc6xiViCJANlTguvg/3ChXysKLOyg2HcMsvzCdtNGLTXPt7Dmop8hza4Do= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHtzmXtZHbwaGAlfGXCePRZVRFqtsvky2Uh/KaRUznrd0dwRXNUcoLDxweo248bA0G7Bq2OXNjBxXqziFyfE3k= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCo8LzbKvqDykYK0VR9mxHzPGyEBOkkYVfQHoXx0g/cifqhd0vc6EpK3+jcuA1Uafw7jMt/oh8ki/6h85YlyyUg= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_07:28:46 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T07:36:35.134 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-17T07:36:35.254 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-17T07:36:35.255 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T07:36:35.256 INFO:teuthology.task.internal:Checking packages... 2024-09-17T07:36:35.278 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-17T07:36:35.278 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T07:36:35.278 INFO:teuthology.packaging:ref: None 2024-09-17T07:36:35.278 INFO:teuthology.packaging:tag: None 2024-09-17T07:36:35.278 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-17T07:36:35.279 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:36:35.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-17T07:36:35.436 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-17T07:36:35.438 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T07:36:35.444 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T07:36:35.444 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T07:36:35.449 INFO:teuthology.task.internal:Saving configuration 2024-09-17T07:36:35.462 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T07:36:35.469 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T07:36:35.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 07:31:41.102736', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbxAaFhqneXjYqjKjSehs56lFBY4Uc6xiViCJANlTguvg/3ChXysKLOyg2HcMsvzCdtNGLTXPt7Dmop8hza4Do='} 2024-09-17T07:36:35.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 07:31:41.101433', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHtzmXtZHbwaGAlfGXCePRZVRFqtsvky2Uh/KaRUznrd0dwRXNUcoLDxweo248bA0G7Bq2OXNjBxXqziFyfE3k='} 2024-09-17T07:36:35.534 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 07:31:41.099935', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCo8LzbKvqDykYK0VR9mxHzPGyEBOkkYVfQHoXx0g/cifqhd0vc6EpK3+jcuA1Uafw7jMt/oh8ki/6h85YlyyUg='} 2024-09-17T07:36:35.534 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T07:36:35.540 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-17T07:36:35.541 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-09-17T07:36:35.541 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['client.0'] 2024-09-17T07:36:35.541 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T07:36:35.622 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec60417ac0>, signals=[15]) 2024-09-17T07:36:35.622 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T07:36:35.628 INFO:teuthology.task.internal:Opening connections... 2024-09-17T07:36:35.628 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:36:35.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:36:35.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:36:35.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:36:35.785 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-09-17T07:36:35.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:36:35.861 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T07:36:35.868 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-09-17T07:36:35.886 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-09-17T07:36:35.886 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-09-17T07:36:35.944 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-09-17T07:36:35.944 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-09-17T07:36:35.944 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-09-17T07:36:35.944 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-09-17T07:36:35.944 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T07:36:35.945 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T07:36:35.946 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-09-17T07:36:35.985 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-09-17T07:36:36.004 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-09-17T07:36:36.004 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-09-17T07:36:36.062 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-09-17T07:36:36.063 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T07:36:36.063 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-09-17T07:36:36.063 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T07:36:36.063 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T07:36:36.063 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T07:36:36.063 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-09-17T07:36:36.110 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-09-17T07:36:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-09-17T07:36:36.128 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-09-17T07:36:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T07:36:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T07:36:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T07:36:36.188 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-09-17T07:36:36.218 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T07:36:36.227 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T07:36:36.232 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T07:36:36.232 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-09-17T07:36:36.235 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-09-17T07:36:36.237 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-09-17T07:36:36.254 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T07:36:36.260 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T07:36:36.260 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-09-17T07:36:36.293 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-09-17T07:36:36.295 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-09-17T07:36:36.322 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T07:36:36.396 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T07:36:36.410 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T07:36:36.410 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T07:36:36.411 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T07:36:36.411 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T07:36:36.412 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T07:36:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:36:36.429 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-09-17T07:36:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:36:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:36:36.431 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-17T07:36:36.431 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-09-17T07:36:36.487 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-503.el9.x86_64 2024-09-17T07:36:36.487 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-503.el9.x86_64 2024-09-17T07:36:36.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-09-17T07:36:36.488 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-503.el9.x86_64 2024-09-17T07:36:36.489 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-503.el9.x86_64 2024-09-17T07:36:36.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-09-17T07:36:36.490 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-503.el9.x86_64 2024-09-17T07:36:36.490 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-503.el9.x86_64 2024-09-17T07:36:36.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-09-17T07:36:38.933 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-17T07:36:40.248 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-17T07:36:41.252 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.3 MB 00:02 2024-09-17T07:36:42.116 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-17T07:36:46.358 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-17T07:36:48.084 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-17T07:36:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-17T07:36:50.185 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-17T07:36:50.325 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 20 MB 00:10 2024-09-17T07:36:54.236 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T07:36:55.719 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-17T07:36:55.983 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-17T07:36:56.151 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T07:36:56.151 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T07:36:56.231 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:36:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-17T07:36:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:36:56.243 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 79 M 2024-09-17T07:36:56.243 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 126 M 2024-09-17T07:36:56.243 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:36:56.637 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 304 kB/s | 36 kB 00:00 2024-09-17T07:36:57.709 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-17T07:36:57.995 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-17T07:36:58.922 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-17T07:36:58.940 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-17T07:37:03.479 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-17T07:37:04.396 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T07:37:04.536 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T07:37:06.344 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T07:37:06.345 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T07:37:06.433 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:37:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-09-17T07:37:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:37:06.441 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 79 M 2024-09-17T07:37:06.441 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 126 M 2024-09-17T07:37:06.441 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:37:06.517 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T07:37:06.517 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T07:37:06.608 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:37:06.613 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:37:06.613 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-09-17T07:37:06.613 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:37:06.613 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-09-17T07:37:06.614 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:37:06.615 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 79 M 2024-09-17T07:37:06.615 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 126 M 2024-09-17T07:37:06.615 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:37:06.698 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 275 kB/s | 36 kB 00:00 2024-09-17T07:37:07.004 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 380 kB/s | 36 kB 00:00 2024-09-17T07:37:07.395 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-17T07:37:07.862 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 36 MB/s | 35 MB 00:00 2024-09-17T07:37:08.121 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-17T07:37:08.121 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:37:08.121 INFO:teuthology.orchestra.run.smithi026.stdout:Total 52 MB/s | 79 MB 00:01 2024-09-17T07:37:08.838 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:37:09.236 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:37:09.236 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:37:10.070 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.1 MB/s | 29 MB 00:13 2024-09-17T07:37:10.189 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 8.2 MB/s | 29 MB 00:03 2024-09-17T07:37:10.225 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:37:10.226 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:37:12.397 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-17T07:37:12.397 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:37:12.397 INFO:teuthology.orchestra.run.smithi098.stdout:Total 4.9 MB/s | 79 MB 00:16 2024-09-17T07:37:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:37:13.081 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:37:13.441 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:37:13.451 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:37:13.451 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:37:13.498 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-17T07:37:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:37:14.411 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:37:14.411 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:37:16.280 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:37:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:37:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:37:17.489 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:37:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:37:20.122 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.5 MB/s | 35 MB 00:13 2024-09-17T07:37:20.123 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:37:20.123 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.8 MB/s | 79 MB 00:13 2024-09-17T07:37:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:37:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:37:20.799 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:37:21.157 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:37:21.158 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:37:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:37:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:37:22.128 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:37:22.128 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:37:24.563 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:37:25.209 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:37:25.305 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:37:25.321 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:37:25.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:37:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:37:28.042 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:37:28.173 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:37:30.191 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:37:33.001 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:37:33.039 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:37:37.868 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:38:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:38:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:38:06.713 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:06.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:38:06.713 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:38:06.714 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:06.714 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T07:38:06.714 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T07:38:06.714 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T07:38:06.714 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:38:06.714 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:38:07.052 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:38:07.135 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:07.135 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:38:07.135 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T07:38:07.136 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T07:38:07.136 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T07:38:07.136 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T07:38:07.136 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T07:38:07.136 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:38:07.195 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:07.195 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:38:07.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 82.el9 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8274670 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-09-17T07:38:07.230 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-09-17T07:38:07.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:38:07.232 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T07:38:07.232 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T07:38:07.232 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-508.el9.x86_64 2024-09-17T07:38:07.233 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T07:38:07.870 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-09-17T07:38:08.466 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:08.505 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:38:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:38:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:38:09.274 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T07:38:09.311 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-09-17T07:38:09.313 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-17T07:38:09.328 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.glMhsk1gqx 2024-09-17T07:38:09.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.glMhsk1gqx 2024-09-17T07:38:09.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.glMhsk1gqx 2024-09-17T07:38:09.538 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.glMhsk1gqx is 6KB 2024-09-17T07:38:09.586 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.glMhsk1gqx 2024-09-17T07:38:09.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T07:38:09.672 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T07:38:09.672 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T07:38:09.672 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T07:38:09.672 INFO:teuthology.orchestra.run.smithi026.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T07:38:09.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T07:38:09.755 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T07:38:09.756 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:38:09.756 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:38:09.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-09-17T07:38:10.101 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:38:10.176 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:10.176 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:38:10.176 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T07:38:10.176 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T07:38:10.176 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T07:38:10.176 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-17T07:38:10.176 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T07:38:10.177 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:38:10.235 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:10.235 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:38:10.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 82.el9 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8274670 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T07:38:10.277 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:38:10.278 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T07:38:10.279 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T07:38:10.279 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-508.el9.x86_64 2024-09-17T07:38:10.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T07:38:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-09-17T07:38:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T07:38:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-09-17T07:38:12.344 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-17T07:38:12.360 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.m59FgLZ7Sx 2024-09-17T07:38:12.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m59FgLZ7Sx 2024-09-17T07:38:12.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.m59FgLZ7Sx 2024-09-17T07:38:12.571 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.m59FgLZ7Sx is 6KB 2024-09-17T07:38:12.619 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.m59FgLZ7Sx 2024-09-17T07:38:12.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T07:38:12.706 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T07:38:12.706 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T07:38:12.706 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T07:38:12.706 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T07:38:12.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T07:38:12.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-09-17T07:38:16.356 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:16.398 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:38:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:38:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:38:17.586 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:38:17.587 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:38:17.987 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:38:18.062 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:18.062 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:38:18.062 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T07:38:18.062 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T07:38:18.062 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T07:38:18.062 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T07:38:18.063 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T07:38:18.063 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:38:18.121 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:38:18.122 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:38:18.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 82.el9 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8274670 2024-09-17T07:38:18.154 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T07:38:18.155 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T07:38:18.156 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-09-17T07:38:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:38:18.156 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T07:38:18.157 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T07:38:18.157 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-508.el9.x86_64 2024-09-17T07:38:18.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T07:38:18.801 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-09-17T07:38:20.143 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T07:38:20.172 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-09-17T07:38:20.183 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-17T07:38:20.199 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.Pcrn77iCTC 2024-09-17T07:38:20.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Pcrn77iCTC 2024-09-17T07:38:20.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.Pcrn77iCTC 2024-09-17T07:38:20.409 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.Pcrn77iCTC is 6KB 2024-09-17T07:38:20.435 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.Pcrn77iCTC 2024-09-17T07:38:20.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T07:38:20.519 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T07:38:20.519 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T07:38:20.519 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T07:38:20.519 INFO:teuthology.orchestra.run.smithi094.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T07:38:20.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T07:38:20.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-09-17T07:38:20.746 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-09-17T07:38:39.849 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T07:38:39.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-17T07:38:39.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:38:42.886 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-17T07:38:42.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-17T07:38:42.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:38:50.681 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T07:38:50.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-17T07:38:50.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:38:58.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-17T07:39:01.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-17T07:39:07.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-17T07:39:07.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:39:09.245 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-09-17T07:39:10.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-17T07:39:10.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:39:13.373 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-17T07:39:18.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-17T07:39:18.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:39:21.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-09-17T07:39:25.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-17T07:39:25.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:39:25.629 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-17T07:39:33.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-17T07:39:33.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:39:37.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-17T07:39:37.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:39:39.760 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-09-17T07:39:40.161 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-17T07:39:40.161 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T07:39:40.161 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-09-17T07:39:40.177 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-508.el9.x86_64 2024-09-17T07:39:40.177 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T07:39:40.177 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T07:39:40.177 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T07:39:41.178 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T07:39:41.179 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T07:39:41.274 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-09-17T07:39:41.294 DEBUG:teuthology.parallel:result is None 2024-09-17T07:39:57.365 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-17T07:39:58.040 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-17T07:39:58.040 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-508.el9.x86_64"... 2024-09-17T07:39:58.040 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-17T07:39:58.055 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-508.el9.x86_64 2024-09-17T07:39:58.055 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T07:39:58.055 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T07:39:58.055 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T07:39:59.056 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-17T07:39:59.057 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T07:39:59.142 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-09-17T07:39:59.161 DEBUG:teuthology.parallel:result is None 2024-09-17T07:40:05.601 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-09-17T07:40:06.096 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-17T07:40:06.096 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T07:40:06.097 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-09-17T07:40:06.118 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-508.el9.x86_64 2024-09-17T07:40:06.118 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T07:40:06.118 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T07:40:06.118 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T07:40:07.119 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T07:40:07.119 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T07:40:07.192 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-09-17T07:40:07.259 DEBUG:teuthology.parallel:result is None 2024-09-17T07:40:07.259 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T07:40:07.269 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T07:40:07.270 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T07:40:07.273 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T07:40:07.275 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T07:40:07.294 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T07:40:07.300 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T07:40:07.307 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T07:40:07.307 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T07:40:07.334 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T07:40:07.336 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T07:40:07.448 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T07:40:07.455 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T07:40:07.455 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T07:40:07.457 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T07:40:07.460 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T07:40:07.488 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:40:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:40:07.493 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:40:07.500 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:40:07.504 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:40:07.504 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:40:07.506 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T07:40:07.513 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T07:40:07.513 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T07:40:07.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T07:40:07.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T07:40:07.575 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T07:40:07.584 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T07:40:07.584 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T07:40:07.615 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T07:40:07.618 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T07:40:07.635 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:40:07.716 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:40:07.764 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:40:07.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:40:07.893 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:40:07.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T07:40:07.961 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:40:08.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:40:08.061 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:40:08.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:40:08.185 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:40:08.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T07:40:08.251 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:40:08.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:40:08.358 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:40:08.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:40:08.444 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:40:08.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T07:40:08.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-09-17T07:40:08.513 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-09-17T07:40:08.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-09-17T07:40:08.561 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:40:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:40:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:40:09.073 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T07:40:09.081 INFO:teuthology.task.internal:Starting timer... 2024-09-17T07:40:09.081 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T07:40:09.093 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T07:40:09.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-09-17T07:40:09.170 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-09-17T07:40:09.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-09-17T07:40:09.223 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-09-17T07:40:09.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-09-17T07:40:09.265 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-09-17T07:40:09.267 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T07:40:09.267 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-09-17T07:40:09.297 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-09-17T07:40:09.297 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-09-17T07:40:09.324 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-09-17T07:40:09.324 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-09-17T07:40:09.352 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-09-17T07:40:09.352 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-09-17T07:40:09.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:40:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:09.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:40:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:09.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:40:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:09.440 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T07:40:09.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-09-17T07:40:09.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-09-17T07:40:09.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-09-17T07:40:09.545 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T07:40:09.554 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T07:40:09.567 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T07:40:09.568 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi094.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T07:43:57.416 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-09-17T07:43:57.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-17T07:43:57.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:43:57.498 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-09-17T07:43:57.577 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-17T07:43:57.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-17T07:43:57.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:43:57.657 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-09-17T07:43:57.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-17T07:43:57.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-17T07:43:57.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:43:57.817 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-17T07:43:57.901 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-17T07:43:57.902 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T07:43:57.912 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T07:43:57.913 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T07:43:57.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:43:57.916 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T07:43:57.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:43:57.918 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T07:43:57.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:43:57.949 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T07:43:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T07:43:57.965 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T07:43:57.966 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T07:43:57.970 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T07:43:57.985 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T07:43:57.991 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-09-17T07:43:57.994 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-09-17T07:43:58.003 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-09-17T07:43:58.006 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-09-17T07:43:58.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-09-17T07:43:58.017 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T07:43:58.020 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-09-17T07:43:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T07:43:58.031 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T07:43:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T07:43:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T07:43:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T07:43:58.095 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:43:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:43:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-09-17T07:43:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:43:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:43:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-09-17T07:43:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:43:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:43:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-17T07:43:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:43:58.115 INFO:teuthology.run_tasks:Running task install... 2024-09-17T07:43:58.123 DEBUG:teuthology.task.install:project ceph 2024-09-17T07:43:58.123 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-17T07:43:58.123 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-17T07:43:58.123 INFO:teuthology.task.install:Using flavor: default 2024-09-17T07:43:58.177 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T07:43:58.177 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-09-17T07:43:58.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo dnf -y install 'dnf-command(copr)' 2024-09-17T07:43:58.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo dnf -y install 'dnf-command(copr)' 2024-09-17T07:43:58.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo dnf -y install 'dnf-command(copr)' 2024-09-17T07:43:58.753 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 07:43:08 AM UTC. 2024-09-17T07:43:58.761 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 07:43:08 AM UTC. 2024-09-17T07:43:58.783 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:46 ago on Tue 17 Sep 2024 07:43:12 AM UTC. 2024-09-17T07:43:58.839 INFO:teuthology.orchestra.run.smithi098.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-17T07:43:58.846 INFO:teuthology.orchestra.run.smithi094.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-17T07:43:58.872 INFO:teuthology.orchestra.run.smithi026.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-17T07:43:58.880 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:43:58.881 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:43:58.882 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:43:58.889 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:43:58.892 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:43:58.892 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:43:58.917 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:43:58.918 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:43:58.918 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:43:58.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo dnf -y copr enable ceph/el9 2024-09-17T07:43:58.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo dnf -y copr enable ceph/el9 2024-09-17T07:43:58.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo dnf -y copr enable ceph/el9 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:of the main distribution, and quality may vary. 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:, 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:and packages are not held to any quality or security level. 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-17T07:43:59.109 INFO:teuthology.orchestra.run.smithi098.stderr:Please do not file bug reports about these packages in Fedora 2024-09-17T07:43:59.110 INFO:teuthology.orchestra.run.smithi098.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:of the main distribution, and quality may vary. 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:, 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:and packages are not held to any quality or security level. 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:Please do not file bug reports about these packages in Fedora 2024-09-17T07:43:59.125 INFO:teuthology.orchestra.run.smithi094.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-17T07:43:59.174 INFO:teuthology.orchestra.run.smithi026.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-17T07:43:59.174 INFO:teuthology.orchestra.run.smithi026.stderr:of the main distribution, and quality may vary. 2024-09-17T07:43:59.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:43:59.174 INFO:teuthology.orchestra.run.smithi026.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-17T07:43:59.175 INFO:teuthology.orchestra.run.smithi026.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-17T07:43:59.175 INFO:teuthology.orchestra.run.smithi026.stderr:, 2024-09-17T07:43:59.175 INFO:teuthology.orchestra.run.smithi026.stderr:and packages are not held to any quality or security level. 2024-09-17T07:43:59.175 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:43:59.175 INFO:teuthology.orchestra.run.smithi026.stderr:Please do not file bug reports about these packages in Fedora 2024-09-17T07:43:59.175 INFO:teuthology.orchestra.run.smithi026.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-17T07:43:59.225 INFO:teuthology.orchestra.run.smithi098.stdout:Repository successfully enabled. 2024-09-17T07:43:59.234 INFO:teuthology.orchestra.run.smithi094.stdout:Repository successfully enabled. 2024-09-17T07:43:59.238 INFO:teuthology.orchestra.run.smithi026.stdout:Repository successfully enabled. 2024-09-17T07:43:59.260 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-17T07:43:59.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:43:59.268 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-17T07:43:59.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:43:59.274 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-17T07:43:59.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:43:59.417 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-17T07:43:59.418 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-17T07:43:59.424 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-17T07:43:59.424 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-17T07:43:59.428 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-17T07:43:59.428 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-17T07:43:59.538 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T07:43:59.539 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:43:59.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T07:43:59.546 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T07:43:59.546 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:43:59.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T07:43:59.550 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T07:43:59.550 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:43:59.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T07:43:59.564 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-09-17T07:43:59.564 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T07:43:59.572 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-09-17T07:43:59.573 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T07:43:59.576 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-09-17T07:43:59.576 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T07:43:59.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T07:43:59.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T07:43:59.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T07:43:59.704 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T07:43:59.717 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T07:43:59.718 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T07:43:59.769 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-09-17T07:43:59.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-09-17T07:43:59.782 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-09-17T07:43:59.783 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-09-17T07:43:59.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-09-17T07:43:59.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-09-17T07:44:00.007 INFO:teuthology.orchestra.run.smithi098.stdout:45 files removed 2024-09-17T07:44:00.017 INFO:teuthology.orchestra.run.smithi094.stdout:45 files removed 2024-09-17T07:44:00.025 INFO:teuthology.orchestra.run.smithi026.stdout:45 files removed 2024-09-17T07:44:00.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-09-17T07:44:00.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-09-17T07:44:00.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-09-17T07:44:00.621 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 256 kB/s | 77 kB 00:00 2024-09-17T07:44:00.626 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 262 kB/s | 77 kB 00:00 2024-09-17T07:44:00.635 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 275 kB/s | 77 kB 00:00 2024-09-17T07:44:00.910 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-17T07:44:00.911 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 47 kB/s | 11 kB 00:00 2024-09-17T07:44:00.913 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-17T07:44:01.170 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 8.1 kB/s | 1.8 kB 00:00 2024-09-17T07:44:01.185 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-17T07:44:01.189 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 7.4 kB/s | 1.8 kB 00:00 2024-09-17T07:44:01.353 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-17T07:44:01.354 INFO:teuthology.orchestra.run.smithi026.stdout:Copr repo for el9 owned by ceph 52 kB/s | 7.6 kB 00:00 2024-09-17T07:44:01.356 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-17T07:44:01.951 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.3 MB 00:00 2024-09-17T07:44:03.015 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.3 MB 00:01 2024-09-17T07:44:04.089 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-17T07:44:04.145 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.3 MB 00:02 2024-09-17T07:44:04.733 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-17T07:44:07.254 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:02 2024-09-17T07:44:08.690 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-17T07:44:10.124 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 128 kB/s | 19 kB 00:00 2024-09-17T07:44:10.992 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-17T07:44:11.036 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T07:44:11.714 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-17T07:44:12.784 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-17T07:44:13.557 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-17T07:44:13.689 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T07:44:14.602 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-17T07:44:16.809 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-17T07:44:19.095 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T07:44:19.220 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:44:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T07:44:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:19.230 INFO:teuthology.orchestra.run.smithi026.stdout:Install 22 Packages 2024-09-17T07:44:19.231 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-09-17T07:44:19.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:19.231 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 58 M 2024-09-17T07:44:19.231 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:44:20.075 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-17T07:44:20.277 INFO:teuthology.orchestra.run.smithi026.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 7.0 MB/s | 5.0 MB 00:00 2024-09-17T07:44:20.335 INFO:teuthology.orchestra.run.smithi026.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 433 kB/s | 25 kB 00:00 2024-09-17T07:44:21.094 INFO:teuthology.orchestra.run.smithi026.stdout:(3/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 15 MB/s | 22 MB 00:01 2024-09-17T07:44:21.136 INFO:teuthology.orchestra.run.smithi026.stdout:(4/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 796 kB/s | 637 kB 00:00 2024-09-17T07:44:21.219 INFO:teuthology.orchestra.run.smithi026.stdout:(5/24): libradosstriper1-17.2.7-1643.gd75f2382. 3.6 MB/s | 458 kB 00:00 2024-09-17T07:44:21.410 INFO:teuthology.orchestra.run.smithi026.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 18 MB/s | 4.9 MB 00:00 2024-09-17T07:44:21.419 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:44:21.432 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:44:21.432 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-17T07:44:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-17T07:44:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:21.435 INFO:teuthology.orchestra.run.smithi098.stdout:Install 22 Packages 2024-09-17T07:44:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-09-17T07:44:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-09-17T07:44:21.436 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:44:21.437 INFO:teuthology.orchestra.run.smithi026.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 208 kB/s | 45 kB 00:00 2024-09-17T07:44:21.470 INFO:teuthology.orchestra.run.smithi026.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.0 MB/s | 120 kB 00:00 2024-09-17T07:44:21.504 INFO:teuthology.orchestra.run.smithi026.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.3 MB/s | 156 kB 00:00 2024-09-17T07:44:21.546 INFO:teuthology.orchestra.run.smithi026.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 4.2 MB/s | 320 kB 00:00 2024-09-17T07:44:21.571 INFO:teuthology.orchestra.run.smithi026.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.3 MB/s | 295 kB 00:00 2024-09-17T07:44:21.605 INFO:teuthology.orchestra.run.smithi026.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.7 MB/s | 100 kB 00:00 2024-09-17T07:44:21.755 INFO:teuthology.orchestra.run.smithi026.stdout:(13/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9 6.6 MB/s | 14 MB 00:02 2024-09-17T07:44:21.839 INFO:teuthology.orchestra.run.smithi026.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 600 kB/s | 160 kB 00:00 2024-09-17T07:44:21.864 INFO:teuthology.orchestra.run.smithi026.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 175 kB/s | 45 kB 00:00 2024-09-17T07:44:21.947 INFO:teuthology.orchestra.run.smithi026.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 508 kB/s | 42 kB 00:00 2024-09-17T07:44:21.989 INFO:teuthology.orchestra.run.smithi026.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-17T07:44:22.048 INFO:teuthology.orchestra.run.smithi026.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-17T07:44:22.073 INFO:teuthology.orchestra.run.smithi026.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 597 kB/s | 49 kB 00:00 2024-09-17T07:44:22.098 INFO:teuthology.orchestra.run.smithi026.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T07:44:22.121 INFO:teuthology.orchestra.run.smithi098.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 13 MB/s | 5.0 MB 00:00 2024-09-17T07:44:22.132 INFO:teuthology.orchestra.run.smithi026.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2024-09-17T07:44:22.191 INFO:teuthology.orchestra.run.smithi026.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-17T07:44:22.280 INFO:teuthology.orchestra.run.smithi098.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 159 kB/s | 25 kB 00:00 2024-09-17T07:44:22.308 INFO:teuthology.orchestra.run.smithi026.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 15 MB/s | 3.1 MB 00:00 2024-09-17T07:44:22.372 INFO:teuthology.orchestra.run.smithi098.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-17T07:44:22.384 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:44:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:22.403 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:44:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T07:44:22.404 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-17T07:44:22.405 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout:Install 22 Packages 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:22.406 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 58 M 2024-09-17T07:44:22.407 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:44:22.456 INFO:teuthology.orchestra.run.smithi098.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-17T07:44:22.691 INFO:teuthology.orchestra.run.smithi026.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 5.3 MB/s | 2.9 MB 00:00 2024-09-17T07:44:22.696 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:44:22.696 INFO:teuthology.orchestra.run.smithi026.stdout:Total 17 MB/s | 58 MB 00:03 2024-09-17T07:44:22.761 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:44:22.797 INFO:teuthology.orchestra.run.smithi098.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-17T07:44:22.799 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:44:22.799 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:44:22.956 INFO:teuthology.orchestra.run.smithi098.stdout:(6/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 12 MB/s | 14 MB 00:01 2024-09-17T07:44:22.981 INFO:teuthology.orchestra.run.smithi098.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 246 kB/s | 45 kB 00:00 2024-09-17T07:44:23.023 INFO:teuthology.orchestra.run.smithi098.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 1.8 MB/s | 120 kB 00:00 2024-09-17T07:44:23.049 INFO:teuthology.orchestra.run.smithi098.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.3 MB/s | 156 kB 00:00 2024-09-17T07:44:23.107 INFO:teuthology.orchestra.run.smithi098.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 3.7 MB/s | 320 kB 00:00 2024-09-17T07:44:23.133 INFO:teuthology.orchestra.run.smithi098.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 3.4 MB/s | 295 kB 00:00 2024-09-17T07:44:23.277 INFO:teuthology.orchestra.run.smithi094.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 6.4 MB/s | 5.0 MB 00:00 2024-09-17T07:44:23.335 INFO:teuthology.orchestra.run.smithi094.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 433 kB/s | 25 kB 00:00 2024-09-17T07:44:23.358 INFO:teuthology.orchestra.run.smithi098.stdout:(12/24): ceph-common-17.2.7-1643.gd75f2382.el9. 14 MB/s | 22 MB 00:01 2024-09-17T07:44:23.383 INFO:teuthology.orchestra.run.smithi098.stdout:(13/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 361 kB/s | 100 kB 00:00 2024-09-17T07:44:23.410 INFO:teuthology.orchestra.run.smithi094.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 8.3 MB/s | 637 kB 00:00 2024-09-17T07:44:23.458 INFO:teuthology.orchestra.run.smithi098.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2024-09-17T07:44:23.476 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:44:23.476 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:44:23.492 INFO:teuthology.orchestra.run.smithi098.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-17T07:44:23.496 INFO:teuthology.orchestra.run.smithi094.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-17T07:44:23.534 INFO:teuthology.orchestra.run.smithi098.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-17T07:44:23.559 INFO:teuthology.orchestra.run.smithi098.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-17T07:44:23.601 INFO:teuthology.orchestra.run.smithi098.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-17T07:44:23.644 INFO:teuthology.orchestra.run.smithi098.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 579 kB/s | 49 kB 00:00 2024-09-17T07:44:23.669 INFO:teuthology.orchestra.run.smithi098.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-17T07:44:23.694 INFO:teuthology.orchestra.run.smithi098.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 746 kB/s | 67 kB 00:00 2024-09-17T07:44:23.761 INFO:teuthology.orchestra.run.smithi098.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-17T07:44:23.836 INFO:teuthology.orchestra.run.smithi094.stdout:(5/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 17 MB/s | 22 MB 00:01 2024-09-17T07:44:23.870 INFO:teuthology.orchestra.run.smithi098.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 15 MB/s | 3.1 MB 00:00 2024-09-17T07:44:23.919 INFO:teuthology.orchestra.run.smithi094.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-09-17T07:44:23.936 INFO:teuthology.orchestra.run.smithi094.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 451 kB/s | 45 kB 00:00 2024-09-17T07:44:23.945 INFO:teuthology.orchestra.run.smithi098.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-09-17T07:44:23.950 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:44:23.951 INFO:teuthology.orchestra.run.smithi098.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-17T07:44:23.970 INFO:teuthology.orchestra.run.smithi094.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.3 MB/s | 120 kB 00:00 2024-09-17T07:44:23.995 INFO:teuthology.orchestra.run.smithi094.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.6 MB/s | 156 kB 00:00 2024-09-17T07:44:24.016 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:44:24.049 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:44:24.050 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:44:24.137 INFO:teuthology.orchestra.run.smithi094.stdout:(10/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9 8.8 MB/s | 14 MB 00:01 2024-09-17T07:44:24.171 INFO:teuthology.orchestra.run.smithi094.stdout:(11/24): python3-rados-17.2.7-1643.gd75f2382.el 1.6 MB/s | 320 kB 00:00 2024-09-17T07:44:24.196 INFO:teuthology.orchestra.run.smithi094.stdout:(12/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 1.4 MB/s | 295 kB 00:00 2024-09-17T07:44:24.213 INFO:teuthology.orchestra.run.smithi094.stdout:(13/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.3 MB/s | 100 kB 00:00 2024-09-17T07:44:24.305 INFO:teuthology.orchestra.run.smithi094.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-17T07:44:24.347 INFO:teuthology.orchestra.run.smithi094.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 912 kB/s | 160 kB 00:00 2024-09-17T07:44:24.397 INFO:teuthology.orchestra.run.smithi094.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-17T07:44:24.448 INFO:teuthology.orchestra.run.smithi094.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-17T07:44:24.473 INFO:teuthology.orchestra.run.smithi094.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-09-17T07:44:24.498 INFO:teuthology.orchestra.run.smithi094.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-17T07:44:24.540 INFO:teuthology.orchestra.run.smithi094.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-17T07:44:24.574 INFO:teuthology.orchestra.run.smithi094.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-09-17T07:44:24.616 INFO:teuthology.orchestra.run.smithi094.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-17T07:44:24.693 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:44:24.693 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:44:24.741 INFO:teuthology.orchestra.run.smithi094.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 18 MB/s | 2.9 MB 00:00 2024-09-17T07:44:24.791 INFO:teuthology.orchestra.run.smithi094.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 12 MB/s | 3.1 MB 00:00 2024-09-17T07:44:24.796 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:44:24.796 INFO:teuthology.orchestra.run.smithi094.stdout:Total 24 MB/s | 58 MB 00:02 2024-09-17T07:44:24.860 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:44:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:44:24.893 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:44:24.893 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:44:25.222 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-17T07:44:25.320 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:25.477 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-17T07:44:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:25.532 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:44:25.533 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:44:25.595 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:25.648 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-17T07:44:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-17T07:44:25.945 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:44:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-17T07:44:26.183 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-17T07:44:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-17T07:44:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-17T07:44:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-17T07:44:26.323 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-17T07:44:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-17T07:44:26.389 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:26.418 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-17T07:44:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-17T07:44:26.481 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-17T07:44:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-17T07:44:26.569 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-17T07:44:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:26.602 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-17T07:44:26.636 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-17T07:44:26.687 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:26.741 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-17T07:44:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:44:26.786 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-17T07:44:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-17T07:44:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-17T07:44:27.097 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-17T07:44:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-17T07:44:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-17T07:44:27.208 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-17T07:44:27.222 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-17T07:44:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:27.252 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-17T07:44:27.303 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-17T07:44:27.351 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-17T07:44:27.389 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-17T07:44:27.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-17T07:44:27.428 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:27.431 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-17T07:44:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-17T07:44:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-17T07:44:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-17T07:44:27.601 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-17T07:44:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-17T07:44:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-17T07:44:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-17T07:44:27.721 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-17T07:44:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-17T07:44:28.038 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-17T07:44:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-17T07:44:28.065 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-17T07:44:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-17T07:44:28.128 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-17T07:44:28.165 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-17T07:44:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-17T07:44:28.208 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-17T07:44:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-17T07:44:28.245 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-17T07:44:28.307 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-17T07:44:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-17T07:44:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-17T07:44:28.465 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-17T07:44:28.501 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-17T07:44:28.535 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-17T07:44:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-17T07:44:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-17T07:44:28.963 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-17T07:44:29.037 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-17T07:44:29.066 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-17T07:44:29.737 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:29.772 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-17T07:44:30.491 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-17T07:44:30.491 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T07:44:30.491 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T07:44:30.491 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:30.526 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:30.640 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:30.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:31.236 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-17T07:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-17T07:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T07:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T07:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:31.371 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:31.533 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:31.572 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:32.098 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-17T07:44:32.133 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-17T07:44:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-17T07:44:32.233 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T07:44:32.233 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T07:44:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:32.268 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:40.740 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:40.740 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-09-17T07:44:40.740 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-09-17T07:44:40.741 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-09-17T07:44:40.741 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-09-17T07:44:40.741 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-09-17T07:44:40.741 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-09-17T07:44:40.741 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-09-17T07:44:40.741 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:41.354 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:41.354 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-09-17T07:44:41.355 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-09-17T07:44:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-17T07:44:42.197 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-17T07:44:42.197 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:44:42.197 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T07:44:42.197 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T07:44:42.197 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T07:44:42.197 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:42.198 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-17T07:44:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-17T07:44:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:42.710 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-17T07:44:42.772 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-17T07:44:42.773 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:44:42.773 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T07:44:42.773 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T07:44:42.773 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T07:44:42.773 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:42.774 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-17T07:44:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-17T07:44:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:43.407 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-17T07:44:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-17T07:44:43.485 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:44:43.485 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T07:44:43.485 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T07:44:43.486 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T07:44:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-17T07:44:43.577 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-17T07:44:43.577 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:43.669 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:43.669 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-17T07:44:43.669 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-17T07:44:43.670 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-17T07:44:43.671 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-17T07:44:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-17T07:44:44.179 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-17T07:44:44.180 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:44.180 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-17T07:44:44.180 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-17T07:44:44.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:44.694 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:44.694 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-09-17T07:44:44.694 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.695 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:44.696 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:44:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-17T07:44:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-17T07:44:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-17T07:44:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-17T07:44:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-17T07:44:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-17T07:44:45.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:45.079 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:45.079 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T07:44:45.080 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:45.081 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:44:45.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-09-17T07:44:45.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-09-17T07:44:45.705 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:29 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:44:45.715 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T07:44:45.716 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:45.717 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T07:44:45.718 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:45.718 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:44:45.841 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:44:45.841 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-09-17T07:44:45.842 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:45.843 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 48 M 2024-09-17T07:44:45.843 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 193 M 2024-09-17T07:44:45.843 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:44:46.057 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:27 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:44:46.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-09-17T07:44:46.187 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:44:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-17T07:44:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-09-17T07:44:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 193 M 2024-09-17T07:44:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:44:46.393 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 560 kB/s | 163 kB 00:00 2024-09-17T07:44:46.536 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 710 kB/s | 163 kB 00:00 2024-09-17T07:44:46.735 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 480 kB/s | 304 kB 00:00 2024-09-17T07:44:46.760 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 173 kB/s | 64 kB 00:00 2024-09-17T07:44:46.762 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:26 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:44:46.894 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T07:44:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T07:44:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:46.896 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:44:46.896 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:46.896 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-09-17T07:44:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:46.896 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 48 M 2024-09-17T07:44:46.897 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 193 M 2024-09-17T07:44:46.897 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:44:46.911 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-17T07:44:46.944 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 476 kB/s | 304 kB 00:00 2024-09-17T07:44:47.049 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-17T07:44:47.083 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-17T07:44:47.150 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 713 kB/s | 163 kB 00:00 2024-09-17T07:44:48.285 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 22 MB/s | 48 MB 00:02 2024-09-17T07:44:48.287 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:44:48.288 INFO:teuthology.orchestra.run.smithi026.stdout:Total 20 MB/s | 48 MB 00:02 2024-09-17T07:44:48.315 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:44:48.337 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:44:48.337 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:44:48.724 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:44:48.724 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:44:48.992 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 23 MB/s | 48 MB 00:02 2024-09-17T07:44:48.994 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:44:48.994 INFO:teuthology.orchestra.run.smithi094.stdout:Total 23 MB/s | 48 MB 00:02 2024-09-17T07:44:49.021 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:44:49.040 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:44:49.040 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:44:49.267 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:44:49.338 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T07:44:49.377 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T07:44:49.404 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-17T07:44:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:44:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:44:49.578 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 15 MB/s | 48 MB 00:03 2024-09-17T07:44:49.580 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:44:49.580 INFO:teuthology.orchestra.run.smithi098.stdout:Total 14 MB/s | 48 MB 00:03 2024-09-17T07:44:49.608 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:44:49.627 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:44:49.627 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:44:49.925 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:44:49.993 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:44:49.993 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:44:50.011 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T07:44:50.060 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T07:44:50.085 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-17T07:44:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:44:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T07:44:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T07:44:50.696 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-17T07:44:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-17T07:44:54.162 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-17T07:44:54.518 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-17T07:44:54.592 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-17T07:44:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-17T07:44:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-17T07:44:56.545 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-17T07:44:56.545 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-17T07:44:56.545 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-17T07:44:56.546 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T07:44:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T07:44:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:56.868 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:44:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T07:44:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T07:44:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:56.869 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:44:56.901 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-17T07:44:56.902 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-17T07:44:56.902 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-17T07:44:56.902 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T07:44:57.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-09-17T07:44:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T07:44:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:57.232 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:44:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T07:44:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T07:44:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:57.233 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:44:57.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-09-17T07:44:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-17T07:44:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-17T07:44:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-17T07:44:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T07:44:57.820 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:41 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:44:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:57.916 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:44:57.962 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:44:57.964 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:57.964 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:44:57.964 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T07:44:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T07:44:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:57.967 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:44:57.968 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:44:57.968 INFO:teuthology.orchestra.run.smithi026.stdout:Install 34 Packages 2024-09-17T07:44:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:44:57.968 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 27 M 2024-09-17T07:44:57.968 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 96 M 2024-09-17T07:44:57.968 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:44:58.100 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:38 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:44:58.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-09-17T07:44:58.234 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:44:58.236 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:44:58.236 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:58.236 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:44:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-17T07:44:58.236 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T07:44:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T07:44:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T07:44:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T07:44:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:58.240 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:44:58.240 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:44:58.240 INFO:teuthology.orchestra.run.smithi094.stdout:Install 34 Packages 2024-09-17T07:44:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:44:58.241 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 27 M 2024-09-17T07:44:58.241 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 96 M 2024-09-17T07:44:58.241 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:44:58.653 INFO:teuthology.orchestra.run.smithi026.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-17T07:44:58.795 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:44:58.862 INFO:teuthology.orchestra.run.smithi026.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.2 MB/s | 2.1 MB 00:00 2024-09-17T07:44:58.903 INFO:teuthology.orchestra.run.smithi026.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.8 MB/s | 1.4 MB 00:00 2024-09-17T07:44:58.944 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:44:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:44:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:44:58.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-17T07:44:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T07:44:58.947 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T07:44:58.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout:Install 34 Packages 2024-09-17T07:44:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:44:58.950 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 27 M 2024-09-17T07:44:58.950 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 96 M 2024-09-17T07:44:58.950 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:44:59.010 INFO:teuthology.orchestra.run.smithi094.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-17T07:44:59.012 INFO:teuthology.orchestra.run.smithi026.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-17T07:44:59.038 INFO:teuthology.orchestra.run.smithi026.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 1.8 MB/s | 240 kB 00:00 2024-09-17T07:44:59.155 INFO:teuthology.orchestra.run.smithi026.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-17T07:44:59.196 INFO:teuthology.orchestra.run.smithi026.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-17T07:44:59.238 INFO:teuthology.orchestra.run.smithi026.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-09-17T07:44:59.263 INFO:teuthology.orchestra.run.smithi026.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-17T07:44:59.289 INFO:teuthology.orchestra.run.smithi026.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-17T07:44:59.302 INFO:teuthology.orchestra.run.smithi094.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.4 MB/s | 1.4 MB 00:00 2024-09-17T07:44:59.314 INFO:teuthology.orchestra.run.smithi026.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-17T07:44:59.369 INFO:teuthology.orchestra.run.smithi094.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 4.3 MB/s | 2.1 MB 00:00 2024-09-17T07:44:59.419 INFO:teuthology.orchestra.run.smithi094.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 4.7 MB/s | 240 kB 00:00 2024-09-17T07:44:59.531 INFO:teuthology.orchestra.run.smithi026.stdout:(12/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 23 MB/s | 15 MB 00:00 2024-09-17T07:44:59.557 INFO:teuthology.orchestra.run.smithi026.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 144 kB/s | 35 kB 00:00 2024-09-17T07:44:59.581 INFO:teuthology.orchestra.run.smithi026.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 589 kB/s | 172 kB 00:00 2024-09-17T07:44:59.596 INFO:teuthology.orchestra.run.smithi098.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 50 kB/s | 6.5 kB 00:00 2024-09-17T07:44:59.607 INFO:teuthology.orchestra.run.smithi026.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-17T07:44:59.632 INFO:teuthology.orchestra.run.smithi026.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 827 kB/s | 42 kB 00:00 2024-09-17T07:44:59.645 INFO:teuthology.orchestra.run.smithi094.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 6.2 MB/s | 3.9 MB 00:00 2024-09-17T07:44:59.670 INFO:teuthology.orchestra.run.smithi094.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-17T07:44:59.674 INFO:teuthology.orchestra.run.smithi026.stdout:(17/34): python3-pytz-2021.1-5.el9.noarch.rpm 434 kB/s | 51 kB 00:00 2024-09-17T07:44:59.716 INFO:teuthology.orchestra.run.smithi026.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 399 kB/s | 43 kB 00:00 2024-09-17T07:44:59.720 INFO:teuthology.orchestra.run.smithi094.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-17T07:44:59.741 INFO:teuthology.orchestra.run.smithi026.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 465 kB/s | 11 kB 00:00 2024-09-17T07:44:59.766 INFO:teuthology.orchestra.run.smithi026.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-17T07:44:59.779 INFO:teuthology.orchestra.run.smithi094.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-17T07:44:59.792 INFO:teuthology.orchestra.run.smithi026.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-17T07:44:59.813 INFO:teuthology.orchestra.run.smithi098.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-09-17T07:44:59.817 INFO:teuthology.orchestra.run.smithi026.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-17T07:44:59.829 INFO:teuthology.orchestra.run.smithi094.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-17T07:44:59.842 INFO:teuthology.orchestra.run.smithi026.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 390 kB/s | 19 kB 00:00 2024-09-17T07:44:59.854 INFO:teuthology.orchestra.run.smithi098.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.6 MB/s | 1.4 MB 00:00 2024-09-17T07:44:59.859 INFO:teuthology.orchestra.run.smithi026.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 470 kB/s | 20 kB 00:00 2024-09-17T07:44:59.876 INFO:teuthology.orchestra.run.smithi026.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-17T07:44:59.888 INFO:teuthology.orchestra.run.smithi094.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-17T07:44:59.901 INFO:teuthology.orchestra.run.smithi026.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-17T07:44:59.935 INFO:teuthology.orchestra.run.smithi026.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-17T07:44:59.952 INFO:teuthology.orchestra.run.smithi026.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-17T07:44:59.955 INFO:teuthology.orchestra.run.smithi094.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 4.0 MB/s | 1.2 MB 00:00 2024-09-17T07:44:59.977 INFO:teuthology.orchestra.run.smithi026.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-17T07:44:59.980 INFO:teuthology.orchestra.run.smithi098.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 1.9 MB/s | 240 kB 00:00 2024-09-17T07:44:59.994 INFO:teuthology.orchestra.run.smithi026.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-17T07:45:00.020 INFO:teuthology.orchestra.run.smithi026.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-17T07:45:00.030 INFO:teuthology.orchestra.run.smithi094.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-17T07:45:00.045 INFO:teuthology.orchestra.run.smithi026.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-09-17T07:45:00.055 INFO:teuthology.orchestra.run.smithi094.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 348 kB/s | 35 kB 00:00 2024-09-17T07:45:00.062 INFO:teuthology.orchestra.run.smithi026.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-09-17T07:45:00.080 INFO:teuthology.orchestra.run.smithi094.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-17T07:45:00.086 INFO:teuthology.orchestra.run.smithi026.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-17T07:45:00.091 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:00.091 INFO:teuthology.orchestra.run.smithi026.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-17T07:45:00.099 INFO:teuthology.orchestra.run.smithi098.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 7.8 MB/s | 3.9 MB 00:00 2024-09-17T07:45:00.114 INFO:teuthology.orchestra.run.smithi094.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-17T07:45:00.140 INFO:teuthology.orchestra.run.smithi094.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-09-17T07:45:00.190 INFO:teuthology.orchestra.run.smithi094.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-09-17T07:45:00.219 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:00.248 INFO:teuthology.orchestra.run.smithi094.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 6.2 MB/s | 358 kB 00:00 2024-09-17T07:45:00.260 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:00.260 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:00.274 INFO:teuthology.orchestra.run.smithi094.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-17T07:45:00.299 INFO:teuthology.orchestra.run.smithi094.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-09-17T07:45:00.324 INFO:teuthology.orchestra.run.smithi094.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-17T07:45:00.349 INFO:teuthology.orchestra.run.smithi094.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-17T07:45:00.372 INFO:teuthology.orchestra.run.smithi098.stdout:(6/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 27 MB/s | 15 MB 00:00 2024-09-17T07:45:00.375 INFO:teuthology.orchestra.run.smithi094.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-09-17T07:45:00.400 INFO:teuthology.orchestra.run.smithi094.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-17T07:45:00.426 INFO:teuthology.orchestra.run.smithi094.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-17T07:45:00.451 INFO:teuthology.orchestra.run.smithi094.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T07:45:00.476 INFO:teuthology.orchestra.run.smithi094.stdout:(27/34): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-17T07:45:00.493 INFO:teuthology.orchestra.run.smithi094.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-09-17T07:45:00.518 INFO:teuthology.orchestra.run.smithi094.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-17T07:45:00.535 INFO:teuthology.orchestra.run.smithi094.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 852 kB/s | 36 kB 00:00 2024-09-17T07:45:00.569 INFO:teuthology.orchestra.run.smithi094.stdout:(31/34): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-17T07:45:00.586 INFO:teuthology.orchestra.run.smithi094.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-17T07:45:00.611 INFO:teuthology.orchestra.run.smithi094.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-17T07:45:00.673 INFO:teuthology.orchestra.run.smithi098.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 366 kB/s | 253 kB 00:00 2024-09-17T07:45:00.703 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:00.703 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:00.770 INFO:teuthology.orchestra.run.smithi094.stdout:(34/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 10 MB/s | 15 MB 00:01 2024-09-17T07:45:00.774 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:00.774 INFO:teuthology.orchestra.run.smithi094.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-17T07:45:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 224 kB/s | 106 kB 00:00 2024-09-17T07:45:00.873 INFO:teuthology.orchestra.run.smithi098.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-17T07:45:00.901 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:00.938 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:00.939 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:00.990 INFO:teuthology.orchestra.run.smithi098.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-17T07:45:01.049 INFO:teuthology.orchestra.run.smithi098.stdout:(11/34): python3-requests-2.25.1-8.el9.noarch.r 622 kB/s | 125 kB 00:00 2024-09-17T07:45:01.157 INFO:teuthology.orchestra.run.smithi098.stdout:(12/34): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-17T07:45:01.174 INFO:teuthology.orchestra.run.smithi098.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-09-17T07:45:01.199 INFO:teuthology.orchestra.run.smithi098.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 824 kB/s | 172 kB 00:00 2024-09-17T07:45:01.233 INFO:teuthology.orchestra.run.smithi098.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-17T07:45:01.258 INFO:teuthology.orchestra.run.smithi098.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-09-17T07:45:01.382 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:01.382 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:01.382 INFO:teuthology.orchestra.run.smithi098.stdout:(17/34): python3-pytz-2021.1-5.el9.noarch.rpm 467 kB/s | 51 kB 00:00 2024-09-17T07:45:01.382 INFO:teuthology.orchestra.run.smithi098.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-17T07:45:01.382 INFO:teuthology.orchestra.run.smithi098.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 340 kB/s | 11 kB 00:00 2024-09-17T07:45:01.393 INFO:teuthology.orchestra.run.smithi098.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-17T07:45:01.418 INFO:teuthology.orchestra.run.smithi098.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-17T07:45:01.444 INFO:teuthology.orchestra.run.smithi098.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-17T07:45:01.469 INFO:teuthology.orchestra.run.smithi098.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-17T07:45:01.494 INFO:teuthology.orchestra.run.smithi098.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-17T07:45:01.519 INFO:teuthology.orchestra.run.smithi098.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-17T07:45:01.545 INFO:teuthology.orchestra.run.smithi098.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T07:45:01.570 INFO:teuthology.orchestra.run.smithi098.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-17T07:45:01.595 INFO:teuthology.orchestra.run.smithi098.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-17T07:45:01.621 INFO:teuthology.orchestra.run.smithi098.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T07:45:01.646 INFO:teuthology.orchestra.run.smithi098.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-17T07:45:01.671 INFO:teuthology.orchestra.run.smithi098.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-17T07:45:01.696 INFO:teuthology.orchestra.run.smithi098.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-17T07:45:01.722 INFO:teuthology.orchestra.run.smithi098.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-09-17T07:45:01.747 INFO:teuthology.orchestra.run.smithi098.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-17T07:45:01.751 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:01.751 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.7 MB/s | 27 MB 00:02 2024-09-17T07:45:01.880 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:01.917 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:01.917 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:02.009 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:02.077 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-17T07:45:02.109 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-17T07:45:02.142 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-17T07:45:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-17T07:45:02.315 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-17T07:45:02.316 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:02.316 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-17T07:45:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-17T07:45:02.426 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-17T07:45:02.461 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-17T07:45:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-17T07:45:02.535 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-17T07:45:02.587 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-17T07:45:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:02.629 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-17T07:45:02.656 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-17T07:45:02.678 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-17T07:45:02.688 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-17T07:45:02.718 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-17T07:45:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-17T07:45:02.795 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-17T07:45:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-17T07:45:02.882 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-17T07:45:02.927 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-17T07:45:02.954 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-17T07:45:02.962 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-17T07:45:02.993 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-17T07:45:03.027 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-17T07:45:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-17T07:45:03.061 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-17T07:45:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-17T07:45:03.101 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-17T07:45:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-17T07:45:03.144 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-17T07:45:03.179 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-17T07:45:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-17T07:45:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-17T07:45:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-17T07:45:03.259 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-17T07:45:03.332 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-17T07:45:03.455 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-17T07:45:03.470 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-17T07:45:03.514 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-17T07:45:03.550 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-17T07:45:03.587 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-17T07:45:03.605 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-17T07:45:03.642 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-17T07:45:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-17T07:45:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-17T07:45:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-17T07:45:03.700 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-17T07:45:03.707 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-17T07:45:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-17T07:45:03.771 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-17T07:45:03.812 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-17T07:45:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-17T07:45:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-17T07:45:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-17T07:45:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-17T07:45:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-17T07:45:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-17T07:45:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-17T07:45:04.032 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-17T07:45:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-17T07:45:04.064 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-17T07:45:04.091 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-17T07:45:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-17T07:45:04.092 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:04.093 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T07:45:04.093 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T07:45:04.093 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T07:45:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:04.096 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-17T07:45:04.137 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-17T07:45:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-17T07:45:04.180 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-17T07:45:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-17T07:45:04.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-17T07:45:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-17T07:45:04.312 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-17T07:45:04.334 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-17T07:45:04.386 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-17T07:45:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-17T07:45:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-17T07:45:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-17T07:45:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-17T07:45:04.538 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:04.538 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T07:45:04.538 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T07:45:04.538 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T07:45:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:04.612 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-17T07:45:04.665 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-17T07:45:04.703 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-17T07:45:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-17T07:45:04.808 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-17T07:45:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-17T07:45:05.058 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-17T07:45:05.153 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-17T07:45:05.242 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-17T07:45:05.282 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-17T07:45:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-17T07:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-17T07:45:05.511 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-17T07:45:05.511 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:05.511 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T07:45:05.511 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T07:45:05.511 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T07:45:05.511 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:05.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-17T07:45:05.581 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-17T07:45:05.608 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-17T07:45:05.608 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:05.608 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T07:45:05.608 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T07:45:05.608 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T07:45:05.608 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-17T07:45:05.916 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-17T07:45:05.941 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-17T07:45:05.941 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:05.941 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T07:45:05.942 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T07:45:05.942 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T07:45:05.942 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-17T07:45:05.944 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:05.944 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T07:45:05.944 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T07:45:05.944 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T07:45:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:06.150 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-17T07:45:06.175 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-17T07:45:06.175 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:06.175 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T07:45:06.175 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T07:45:06.175 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T07:45:06.175 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:06.241 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-17T07:45:06.323 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-17T07:45:06.347 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-17T07:45:06.347 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:06.348 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T07:45:06.348 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T07:45:06.348 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T07:45:06.348 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:06.549 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-17T07:45:06.573 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-17T07:45:06.573 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:06.573 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T07:45:06.573 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T07:45:06.574 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T07:45:06.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:06.637 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-17T07:45:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-17T07:45:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-17T07:45:06.943 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:06.944 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T07:45:06.944 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T07:45:06.944 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T07:45:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:07.325 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-17T07:45:07.349 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-17T07:45:07.349 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:07.349 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T07:45:07.349 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T07:45:07.349 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T07:45:07.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:07.551 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-17T07:45:07.574 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-17T07:45:07.574 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:07.574 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T07:45:07.574 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T07:45:07.575 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T07:45:07.575 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:07.647 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-17T07:45:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-17T07:45:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-17T07:45:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-17T07:45:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-17T07:45:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-17T07:45:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-17T07:45:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-17T07:45:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-17T07:45:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-17T07:45:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-17T07:45:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-17T07:45:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-17T07:45:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-17T07:45:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-17T07:45:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-17T07:45:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-17T07:45:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-17T07:45:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-17T07:45:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-17T07:45:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-17T07:45:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T07:45:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T07:45:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:08.830 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T07:45:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T07:45:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:09.186 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:09.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-09-17T07:45:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-17T07:45:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-17T07:45:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-17T07:45:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-17T07:45:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-17T07:45:09.553 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-09-17T07:45:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:09.925 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:09.969 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:09.972 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:45:09.972 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:10.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-09-17T07:45:10.133 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:10.222 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:10.264 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:10.265 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:10.265 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:10.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-09-17T07:45:10.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-17T07:45:10.476 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:10.476 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:10.476 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T07:45:10.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T07:45:10.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:10.479 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:10.646 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:10.783 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:10.784 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:10.785 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:45:10.785 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:10.785 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 83 k 2024-09-17T07:45:10.785 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 375 k 2024-09-17T07:45:10.785 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:10.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-09-17T07:45:10.904 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:10.974 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 437 kB/s | 83 kB 00:00 2024-09-17T07:45:10.974 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:10.975 INFO:teuthology.orchestra.run.smithi026.stdout:Total 434 kB/s | 83 kB 00:00 2024-09-17T07:45:10.975 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:10.980 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:10.980 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:10.982 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:10.982 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:11.033 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:11.034 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:11.035 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:11.035 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 83 k 2024-09-17T07:45:11.035 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 375 k 2024-09-17T07:45:11.035 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:11.065 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:11.232 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 420 kB/s | 83 kB 00:00 2024-09-17T07:45:11.232 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:11.233 INFO:teuthology.orchestra.run.smithi094.stdout:Total 417 kB/s | 83 kB 00:00 2024-09-17T07:45:11.233 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:11.238 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:11.238 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:11.240 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:11.240 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:11.428 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:11.465 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:11.560 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:11.564 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:11.564 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:11.572 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:11.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-09-17T07:45:11.918 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:11.919 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:12.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T07:45:12.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:12.450 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:12.450 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:12.450 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:12.450 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:12.450 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:12.618 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:12.619 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:12.619 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:12.619 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:12.619 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:12.619 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 83 k 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 375 k 2024-09-17T07:45:12.620 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:12.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T07:45:12.818 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 419 kB/s | 83 kB 00:00 2024-09-17T07:45:12.818 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:12.818 INFO:teuthology.orchestra.run.smithi098.stdout:Total 415 kB/s | 83 kB 00:00 2024-09-17T07:45:12.819 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:12.824 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:12.824 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:12.826 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:12.826 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:13.015 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:13.150 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:13.150 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:13.150 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-17T07:45:13.150 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 137 k 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 379 k 2024-09-17T07:45:13.151 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:13.259 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:13.363 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 651 kB/s | 137 kB 00:00 2024-09-17T07:45:13.363 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:13.364 INFO:teuthology.orchestra.run.smithi026.stdout:Total 646 kB/s | 137 kB 00:00 2024-09-17T07:45:13.364 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:13.372 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:13.372 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-17T07:45:13.393 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:13.394 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:13.394 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:13.394 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:13.394 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:13.394 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 137 k 2024-09-17T07:45:13.394 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 379 k 2024-09-17T07:45:13.395 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:13.432 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:13.432 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:13.605 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 655 kB/s | 137 kB 00:00 2024-09-17T07:45:13.605 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:13.606 INFO:teuthology.orchestra.run.smithi094.stdout:Total 649 kB/s | 137 kB 00:00 2024-09-17T07:45:13.606 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:13.613 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:13.613 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:13.655 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:13.669 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:13.669 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:13.682 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:13.682 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:13.682 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T07:45:13.682 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:13.914 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:13.939 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:13.939 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:13.939 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T07:45:13.939 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:14.040 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:14.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:14.041 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:14.041 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:14.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:14.041 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:14.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T07:45:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:14.653 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:14.654 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:14.838 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:14.847 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:14.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:14.847 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:14.847 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:14.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:14.848 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:14.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-09-17T07:45:14.973 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:14.973 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 137 k 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 379 k 2024-09-17T07:45:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:15.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-09-17T07:45:15.183 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 659 kB/s | 137 kB 00:00 2024-09-17T07:45:15.184 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:15.184 INFO:teuthology.orchestra.run.smithi098.stdout:Total 654 kB/s | 137 kB 00:00 2024-09-17T07:45:15.184 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:15.191 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:15.191 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:15.248 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:15.459 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:15.461 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:15.484 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:15.484 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T07:45:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:15.553 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:15.599 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:45:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:15.647 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:15.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T07:45:15.738 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:15.781 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:15.782 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:15.782 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:15.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T07:45:16.274 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:16.407 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-17T07:45:16.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:16.407 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:16.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:16.410 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:16.411 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-17T07:45:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout:Install 7 Packages 2024-09-17T07:45:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:16.414 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.7 M 2024-09-17T07:45:16.414 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 80 M 2024-09-17T07:45:16.414 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:16.431 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:16.563 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:16.563 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T07:45:16.564 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout:Install 7 Packages 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:16.565 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.7 M 2024-09-17T07:45:16.566 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 80 M 2024-09-17T07:45:16.566 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:16.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-09-17T07:45:16.716 INFO:teuthology.orchestra.run.smithi026.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 161 kB/s | 24 kB 00:00 2024-09-17T07:45:16.746 INFO:teuthology.orchestra.run.smithi026.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 83 kB/s | 15 kB 00:00 2024-09-17T07:45:16.787 INFO:teuthology.orchestra.run.smithi026.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-17T07:45:16.800 INFO:teuthology.orchestra.run.smithi026.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 699 kB/s | 41 kB 00:00 2024-09-17T07:45:16.817 INFO:teuthology.orchestra.run.smithi026.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 916 kB/s | 31 kB 00:00 2024-09-17T07:45:16.842 INFO:teuthology.orchestra.run.smithi026.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-17T07:45:16.868 INFO:teuthology.orchestra.run.smithi094.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 96 kB/s | 15 kB 00:00 2024-09-17T07:45:16.893 INFO:teuthology.orchestra.run.smithi094.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1643.gd75 138 kB/s | 24 kB 00:00 2024-09-17T07:45:16.935 INFO:teuthology.orchestra.run.smithi094.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-17T07:45:16.961 INFO:teuthology.orchestra.run.smithi094.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 610 kB/s | 41 kB 00:00 2024-09-17T07:45:16.967 INFO:teuthology.orchestra.run.smithi026.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.3 MB/s | 3.4 MB 00:00 2024-09-17T07:45:16.971 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:16.971 INFO:teuthology.orchestra.run.smithi026.stdout:Total 6.5 MB/s | 3.7 MB 00:00 2024-09-17T07:45:16.971 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:16.980 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:16.980 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:16.985 INFO:teuthology.orchestra.run.smithi094.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 610 kB/s | 31 kB 00:00 2024-09-17T07:45:17.011 INFO:teuthology.orchestra.run.smithi094.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-17T07:45:17.073 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:17.073 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:17.111 INFO:teuthology.orchestra.run.smithi094.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.5 MB/s | 3.4 MB 00:00 2024-09-17T07:45:17.114 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:17.114 INFO:teuthology.orchestra.run.smithi094.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2024-09-17T07:45:17.114 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:17.123 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:17.123 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:17.191 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:17.205 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:17.205 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:17.282 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:17.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T07:45:17.452 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-17T07:45:17.487 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-17T07:45:17.517 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:17.543 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-17T07:45:17.571 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-17T07:45:17.582 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-17T07:45:17.604 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-17T07:45:17.619 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-17T07:45:17.659 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-17T07:45:17.691 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-17T07:45:17.728 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-17T07:45:17.963 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:18.093 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:18.094 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:18.094 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:18.095 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-17T07:45:18.095 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:45:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-17T07:45:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-17T07:45:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-17T07:45:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T07:45:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T07:45:18.096 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-09-17T07:45:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-17T07:45:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:18.097 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:18.097 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:18.097 INFO:teuthology.orchestra.run.smithi098.stdout:Install 7 Packages 2024-09-17T07:45:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:18.098 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.7 M 2024-09-17T07:45:18.098 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 80 M 2024-09-17T07:45:18.098 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:18.395 INFO:teuthology.orchestra.run.smithi098.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 163 kB/s | 24 kB 00:00 2024-09-17T07:45:18.421 INFO:teuthology.orchestra.run.smithi098.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 83 kB/s | 15 kB 00:00 2024-09-17T07:45:18.462 INFO:teuthology.orchestra.run.smithi098.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-17T07:45:18.496 INFO:teuthology.orchestra.run.smithi098.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 966 kB/s | 31 kB 00:00 2024-09-17T07:45:18.521 INFO:teuthology.orchestra.run.smithi098.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 406 kB/s | 41 kB 00:00 2024-09-17T07:45:18.547 INFO:teuthology.orchestra.run.smithi098.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-17T07:45:18.647 INFO:teuthology.orchestra.run.smithi098.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.4 MB/s | 3.4 MB 00:00 2024-09-17T07:45:18.650 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:18.650 INFO:teuthology.orchestra.run.smithi098.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2024-09-17T07:45:18.651 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:18.659 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:18.659 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:18.670 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-17T07:45:18.714 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-17T07:45:18.718 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-17T07:45:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-17T07:45:18.762 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:18.762 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-17T07:45:19.165 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-17T07:45:19.212 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-17T07:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-17T07:45:19.281 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-17T07:45:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-17T07:45:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-17T07:45:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-17T07:45:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-17T07:45:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-17T07:45:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-17T07:45:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-17T07:45:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-17T07:45:19.986 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:19.987 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:20.178 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-17T07:45:20.178 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:20.178 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:20.179 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:20.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T07:45:20.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-17T07:45:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-17T07:45:20.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T07:45:20.789 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:04 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:20.928 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:20.929 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-09-17T07:45:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:20.929 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-09-17T07:45:20.929 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-17T07:45:20.929 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T07:45:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T07:45:20.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T07:45:20.931 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:20.931 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:20.931 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-09-17T07:45:20.931 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-09-17T07:45:20.931 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:20.932 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 38 M 2024-09-17T07:45:20.932 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 212 M 2024-09-17T07:45:20.932 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:20.999 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:21.129 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:21.130 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-09-17T07:45:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:21.130 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-09-17T07:45:21.130 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T07:45:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:21.132 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 38 M 2024-09-17T07:45:21.133 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 212 M 2024-09-17T07:45:21.133 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-17T07:45:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-17T07:45:21.675 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 12 MB/s | 7.4 MB 00:00 2024-09-17T07:45:21.700 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 309 kB/s | 191 kB 00:00 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-17T07:45:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T07:45:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T07:45:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:21.715 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:21.719 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 901 kB/s | 191 kB 00:00 2024-09-17T07:45:21.778 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-17T07:45:21.801 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-17T07:45:21.803 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 33 kB 00:00 2024-09-17T07:45:21.837 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 546 kB/s | 18 kB 00:00 2024-09-17T07:45:21.842 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-09-17T07:45:21.884 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 921 kB/s | 38 kB 00:00 2024-09-17T07:45:21.887 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 769 kB/s | 38 kB 00:00 2024-09-17T07:45:21.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T07:45:22.004 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-17T07:45:22.009 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 864 kB/s | 801 kB 00:00 2024-09-17T07:45:22.071 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-09-17T07:45:22.178 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-17T07:45:22.188 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 5.1 MB 00:00 2024-09-17T07:45:22.263 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-17T07:45:22.289 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 861 kB/s | 245 kB 00:00 2024-09-17T07:45:22.314 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-17T07:45:22.346 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-09-17T07:45:22.429 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-17T07:45:22.523 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:03 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:22.564 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 54 MB/s | 16 MB 00:00 2024-09-17T07:45:22.565 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:22.565 INFO:teuthology.orchestra.run.smithi094.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-17T07:45:22.655 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T07:45:22.656 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-09-17T07:45:22.657 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:22.658 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 38 M 2024-09-17T07:45:22.658 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 212 M 2024-09-17T07:45:22.658 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:22.805 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.4 MB/s | 4.9 MB 00:00 2024-09-17T07:45:22.920 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:22.947 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:22.947 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:22.981 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-17T07:45:23.022 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-17T07:45:23.048 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-09-17T07:45:23.090 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-09-17T07:45:23.157 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 22 MB/s | 3.0 MB 00:00 2024-09-17T07:45:23.182 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 418 kB/s | 38 kB 00:00 2024-09-17T07:45:23.238 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.8 MB/s | 5.1 MB 00:01 2024-09-17T07:45:23.282 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 39 MB/s | 4.9 MB 00:00 2024-09-17T07:45:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-17T07:45:23.424 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): ceph-mgr-diskprediction-local-17.2.7-16 12 MB/s | 7.4 MB 00:00 2024-09-17T07:45:23.443 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:23.444 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:23.449 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.7 MB/s | 250 kB 00:00 2024-09-17T07:45:23.542 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-17T07:45:23.892 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-17T07:45:23.893 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:23.893 INFO:teuthology.orchestra.run.smithi098.stdout:Total 31 MB/s | 38 MB 00:01 2024-09-17T07:45:24.199 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:24.248 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:24.274 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:24.274 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:24.278 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T07:45:24.327 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T07:45:24.432 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T07:45:24.472 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T07:45:24.547 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.7 MB/s | 16 MB 00:02 2024-09-17T07:45:24.547 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:24.547 INFO:teuthology.orchestra.run.smithi026.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-17T07:45:24.656 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T07:45:24.687 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T07:45:24.757 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:24.757 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T07:45:24.916 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:24.943 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:24.943 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T07:45:25.464 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:25.464 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:25.527 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T07:45:25.543 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:25.612 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T07:45:25.667 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T07:45:25.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T07:45:25.808 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T07:45:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T07:45:26.023 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T07:45:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T07:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T07:45:26.394 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T07:45:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T07:45:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T07:45:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T07:45:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T07:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T07:45:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T07:45:26.849 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T07:45:26.895 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T07:45:27.533 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T07:45:27.654 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T07:45:27.839 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T07:45:27.863 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-17T07:45:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T07:45:28.989 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T07:45:29.203 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T07:45:29.232 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-17T07:45:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T07:45:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-17T07:45:30.417 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-17T07:45:30.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-17T07:45:30.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T07:45:30.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T07:45:30.417 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T07:45:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T07:45:30.863 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T07:45:30.863 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:30.863 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:30.863 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:30.863 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:30.864 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:31.269 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-09-17T07:45:31.857 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:11 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:31.991 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:31.992 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:31.992 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:31.992 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T07:45:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T07:45:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:31.994 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:31.994 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:31.994 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-09-17T07:45:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:31.994 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.2 M 2024-09-17T07:45:31.995 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-09-17T07:45:31.995 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-17T07:45:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-17T07:45:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T07:45:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T07:45:32.147 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 339 kB/s | 26 kB 00:00 2024-09-17T07:45:32.172 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 190 kB/s | 19 kB 00:00 2024-09-17T07:45:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-17T07:45:32.273 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-17T07:45:32.298 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 210 kB/s | 47 kB 00:00 2024-09-17T07:45:32.323 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T07:45:32.349 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-17T07:45:32.399 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 319 kB/s | 32 kB 00:00 2024-09-17T07:45:32.425 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-17T07:45:32.450 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T07:45:32.475 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-17T07:45:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T07:45:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:32.491 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T07:45:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T07:45:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:32.493 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:32.500 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-17T07:45:32.534 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-17T07:45:32.537 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:32.537 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.1 MB/s | 2.2 MB 00:00 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T07:45:32.562 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T07:45:32.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T07:45:32.563 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T07:45:32.600 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:32.615 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:32.616 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:32.703 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:32.704 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:32.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T07:45:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T07:45:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:33.036 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:33.203 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:33.371 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T07:45:33.426 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T07:45:33.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-09-17T07:45:33.482 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T07:45:33.493 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:14 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:33.519 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T07:45:33.561 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T07:45:33.624 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:33.625 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T07:45:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T07:45:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-09-17T07:45:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.2 M 2024-09-17T07:45:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-09-17T07:45:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:33.662 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T07:45:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T07:45:33.812 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T07:45:34.043 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:18 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:34.085 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 282 kB/s | 47 kB 00:00 2024-09-17T07:45:34.183 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:45:34.184 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:34.185 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:34.186 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-09-17T07:45:34.186 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:34.186 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.2 M 2024-09-17T07:45:34.186 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-09-17T07:45:34.186 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:34.202 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-09-17T07:45:34.227 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 82 kB/s | 26 kB 00:00 2024-09-17T07:45:34.477 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 316 kB/s | 26 kB 00:00 2024-09-17T07:45:34.502 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 180 kB/s | 19 kB 00:00 2024-09-17T07:45:34.529 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T07:45:34.569 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-17T07:45:34.577 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T07:45:34.577 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-17T07:45:34.594 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-09-17T07:45:34.610 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T07:45:34.619 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 212 kB/s | 47 kB 00:00 2024-09-17T07:45:34.627 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 410 kB/s | 222 kB 00:00 2024-09-17T07:45:34.645 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T07:45:34.656 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T07:45:34.669 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 771 kB/s | 32 kB 00:00 2024-09-17T07:45:34.670 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-17T07:45:34.680 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-17T07:45:34.694 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 458 kB/s | 54 kB 00:00 2024-09-17T07:45:34.720 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 566 kB/s | 279 kB 00:00 2024-09-17T07:45:34.721 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-17T07:45:34.745 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 187 kB/s | 14 kB 00:00 2024-09-17T07:45:34.746 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-17T07:45:34.771 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T07:45:34.796 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-17T07:45:34.804 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-17T07:45:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-17T07:45:34.829 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 775 kB/s | 59 kB 00:00 2024-09-17T07:45:34.854 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-17T07:45:34.872 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.9 MB/s | 1.0 MB 00:00 2024-09-17T07:45:34.875 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:34.875 INFO:teuthology.orchestra.run.smithi026.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-17T07:45:34.896 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-09-17T07:45:34.900 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:34.900 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-17T07:45:34.940 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:34.956 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:34.957 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:34.985 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:34.985 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:35.055 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:35.057 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:35.074 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:35.075 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-17T07:45:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-17T07:45:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T07:45:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T07:45:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T07:45:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T07:45:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T07:45:35.574 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T07:45:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T07:45:35.814 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T07:45:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T07:45:35.885 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T07:45:35.892 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:35.893 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T07:45:35.915 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T07:45:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T07:45:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T07:45:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T07:45:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T07:45:36.089 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T07:45:36.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T07:45:36.167 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T07:45:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T07:45:36.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T07:45:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T07:45:36.720 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:16 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:36.852 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:36.853 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-09-17T07:45:36.854 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:36.855 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 7.0 M 2024-09-17T07:45:36.855 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-09-17T07:45:36.855 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T07:45:36.964 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T07:45:36.998 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T07:45:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T07:45:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T07:45:37.065 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T07:45:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-17T07:45:37.098 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T07:45:37.145 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T07:45:37.167 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-17T07:45:37.471 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-17T07:45:37.529 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T07:45:37.554 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 524 kB/s | 124 kB 00:00 2024-09-17T07:45:37.580 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-17T07:45:37.622 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-17T07:45:37.713 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-17T07:45:37.716 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:37.717 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.1 MB/s | 7.0 MB 00:00 2024-09-17T07:45:37.789 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:37.800 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:37.800 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T07:45:37.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T07:45:37.848 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T07:45:37.848 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T07:45:37.848 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T07:45:37.848 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T07:45:37.848 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T07:45:37.899 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:37.899 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T07:45:37.981 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T07:45:37.982 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T07:45:37.982 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T07:45:37.982 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T07:45:37.982 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T07:45:37.982 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T07:45:38.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T07:45:38.169 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T07:45:38.170 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:38.171 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:38.199 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T07:45:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:38.310 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:38.387 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T07:45:38.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T07:45:38.440 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T07:45:38.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T07:45:38.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T07:45:38.992 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:19 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T07:45:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T07:45:39.123 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:45:39.124 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-09-17T07:45:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:39.126 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 7.0 M 2024-09-17T07:45:39.126 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-09-17T07:45:39.126 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-17T07:45:39.165 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:23 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:39.304 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:39.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:39.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:39.305 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-09-17T07:45:39.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:39.307 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 7.0 M 2024-09-17T07:45:39.307 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-09-17T07:45:39.307 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:39.532 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 564 kB/s | 124 kB 00:00 2024-09-17T07:45:39.616 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-17T07:45:39.641 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T07:45:39.666 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-17T07:45:39.840 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 555 kB/s | 124 kB 00:00 2024-09-17T07:45:39.875 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 441 kB/s | 248 kB 00:00 2024-09-17T07:45:39.899 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T07:45:39.915 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-17T07:45:39.941 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-17T07:45:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-17T07:45:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-17T07:45:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T07:45:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T07:45:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T07:45:39.997 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T07:45:40.275 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:40.276 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:40.333 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 347 kB/s | 248 kB 00:00 2024-09-17T07:45:40.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-09-17T07:45:41.106 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:21 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:41.236 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T07:45:41.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 893 k 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.6 M 2024-09-17T07:45:41.238 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:41.533 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 406 kB/s | 80 kB 00:00 2024-09-17T07:45:41.608 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-17T07:45:41.609 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:41.609 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.3 MB/s | 893 kB 00:00 2024-09-17T07:45:41.621 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:41.643 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:41.643 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:41.721 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:41.722 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:41.907 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T07:45:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-17T07:45:42.828 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-17T07:45:42.828 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-17T07:45:43.075 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.6 MB/s | 6.0 MB 00:03 2024-09-17T07:45:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.8 MB/s | 7.0 MB 00:03 2024-09-17T07:45:43.127 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T07:45:43.127 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:43.127 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:43.127 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T07:45:43.127 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:43.127 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:43.152 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:43.164 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:43.263 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:43.263 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:43.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-09-17T07:45:43.568 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:43.599 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.5 MB/s | 6.0 MB 00:03 2024-09-17T07:45:43.601 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:43.602 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.6 MB/s | 7.0 MB 00:04 2024-09-17T07:45:43.678 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:43.733 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T07:45:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T07:45:43.796 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:43.796 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:44.008 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:24 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:44.121 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:44.141 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:44.142 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:44.143 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:44.143 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:44.143 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:44.143 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 270 k 2024-09-17T07:45:44.143 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.3 M 2024-09-17T07:45:44.143 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:44.290 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T07:45:44.317 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T07:45:44.369 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T07:45:44.389 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-17T07:45:44.389 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:44.390 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-17T07:45:44.390 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T07:45:44.398 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:44.398 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:44.423 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:44.424 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:44.452 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T07:45:44.481 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-17T07:45:44.633 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:44.707 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:44.723 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:44.724 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:44.724 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T07:45:44.724 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T07:45:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T07:45:45.087 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T07:45:45.121 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-17T07:45:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-17T07:45:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-17T07:45:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T07:45:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T07:45:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T07:45:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T07:45:45.503 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T07:45:45.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:45.504 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:45.727 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:45.728 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:45.728 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:45.728 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:45.728 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:45.728 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:45.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-09-17T07:45:45.879 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-17T07:45:45.879 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-17T07:45:45.879 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T07:45:45.879 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T07:45:45.880 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T07:45:45.880 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T07:45:45.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-09-17T07:45:46.148 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:46.149 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:46.355 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:27 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:46.382 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-09-17T07:45:46.486 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:46.486 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-09-17T07:45:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:46.488 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 893 k 2024-09-17T07:45:46.488 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.6 M 2024-09-17T07:45:46.488 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:46.575 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:26 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:46.711 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:46.712 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:46.712 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-17T07:45:46.712 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:46.712 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 126 k 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 448 k 2024-09-17T07:45:46.713 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:46.819 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-17T07:45:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 587 kB/s | 126 kB 00:00 2024-09-17T07:45:46.929 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:46.930 INFO:teuthology.orchestra.run.smithi094.stdout:Total 584 kB/s | 126 kB 00:00 2024-09-17T07:45:46.930 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:46.934 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:46.934 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:46.990 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:46.997 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:46.997 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:47.129 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:47.131 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:47.131 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:47.131 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:47.131 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:47.132 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-17T07:45:47.132 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-09-17T07:45:47.132 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T07:45:47.132 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:47.132 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:47.132 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:47.133 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-09-17T07:45:47.133 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:47.133 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 893 k 2024-09-17T07:45:47.133 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.6 M 2024-09-17T07:45:47.133 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:47.159 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:47.186 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 123 kB/s | 80 kB 00:00 2024-09-17T07:45:47.187 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:47.187 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.2 MB/s | 893 kB 00:00 2024-09-17T07:45:47.199 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:47.221 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:47.221 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:47.250 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:47.300 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:47.301 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:47.485 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:47.500 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-17T07:45:47.630 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T07:45:47.692 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 173 kB/s | 80 kB 00:00 2024-09-17T07:45:47.693 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-17T07:45:47.693 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:47.693 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-09-17T07:45:47.706 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:47.712 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:47.729 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:47.729 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:47.813 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:47.813 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:47.988 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:47.988 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:48.004 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T07:45:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-17T07:45:48.251 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-09-17T07:45:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-17T07:45:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-17T07:45:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T07:45:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:48.620 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T07:45:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:48.620 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:48.839 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:28 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:48.905 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-17T07:45:48.905 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-17T07:45:48.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-09-17T07:45:48.930 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:48.973 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:48.974 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:48.975 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:49.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-09-17T07:45:49.237 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T07:45:49.237 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:49.237 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:49.237 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T07:45:49.237 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:49.237 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:49.494 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:49.550 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-volume 2024-09-17T07:45:49.629 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:49.630 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:49.631 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:49.631 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:45:49.631 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:49.631 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 270 k 2024-09-17T07:45:49.631 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.3 M 2024-09-17T07:45:49.631 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:49.640 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:29 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:49.772 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 25 k 2024-09-17T07:45:49.773 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 110 k 2024-09-17T07:45:49.774 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:49.866 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-17T07:45:49.867 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:49.867 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-17T07:45:49.867 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:49.876 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:49.876 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:49.907 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:49.907 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:49.933 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 157 kB/s | 25 kB 00:00 2024-09-17T07:45:49.933 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:49.933 INFO:teuthology.orchestra.run.smithi094.stdout:Total 156 kB/s | 25 kB 00:00 2024-09-17T07:45:49.933 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:49.937 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:49.937 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:49.938 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:49.939 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:50.111 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:50.129 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:50.164 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:50.192 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:50.193 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T07:45:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:50.302 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 270 k 2024-09-17T07:45:50.303 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.3 M 2024-09-17T07:45:50.304 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:50.527 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:50.538 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-17T07:45:50.539 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:50.539 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-17T07:45:50.539 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:50.548 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:50.549 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:50.581 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:50.581 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:50.747 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:50.748 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:50.748 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:50.748 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:50.748 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:50.748 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:50.883 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:50.883 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:50.883 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T07:45:50.883 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:50.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-09-17T07:45:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:51.211 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:51.211 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:51.504 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:51.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-09-17T07:45:51.594 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:51.707 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-09-17T07:45:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-17T07:45:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:51.914 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-17T07:45:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:51.915 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:33 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:52.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-09-17T07:45:52.263 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:52.264 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:52.264 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-17T07:45:52.264 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:52.264 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:52.265 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-17T07:45:52.265 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:52.265 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:52.265 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:52.266 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:45:52.266 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:52.266 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 126 k 2024-09-17T07:45:52.266 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 448 k 2024-09-17T07:45:52.266 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:52.279 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:32 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:52.368 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:52.411 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:52.412 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:52.412 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:52.473 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 607 kB/s | 126 kB 00:00 2024-09-17T07:45:52.473 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:52.473 INFO:teuthology.orchestra.run.smithi098.stdout:Total 603 kB/s | 126 kB 00:00 2024-09-17T07:45:52.474 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:52.479 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:52.479 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:52.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-09-17T07:45:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:52.542 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:52.795 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:52.838 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:52.975 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 126 k 2024-09-17T07:45:52.976 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 448 k 2024-09-17T07:45:52.977 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:53.066 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:33 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:53.154 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:53.184 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 608 kB/s | 126 kB 00:00 2024-09-17T07:45:53.185 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:53.185 INFO:teuthology.orchestra.run.smithi026.stdout:Total 604 kB/s | 126 kB 00:00 2024-09-17T07:45:53.185 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:53.191 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:53.191 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:53.196 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:53.197 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:53.197 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:53.204 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:53.257 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:53.258 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:53.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-09-17T07:45:53.412 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:53.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-09-17T07:45:53.842 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:33 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:53.931 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:53.950 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:53.973 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:53.974 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:53.974 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:54.052 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-09-17T07:45:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:54.226 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:54.226 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:54.334 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:54.431 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:54.475 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:54.476 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:54.476 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:54.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-09-17T07:45:54.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-09-17T07:45:54.625 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:54.715 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:54.757 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:54.759 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:54.759 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:54.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-09-17T07:45:55.087 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:55.130 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:55.179 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:55.225 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:55.226 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:45:55.226 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:55.262 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:45:55.263 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:45:55.264 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:45:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:55.264 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 25 k 2024-09-17T07:45:55.264 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 110 k 2024-09-17T07:45:55.264 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:45:55.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-09-17T07:45:55.400 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:55.418 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 162 kB/s | 25 kB 00:00 2024-09-17T07:45:55.419 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:55.419 INFO:teuthology.orchestra.run.smithi098.stdout:Total 160 kB/s | 25 kB 00:00 2024-09-17T07:45:55.419 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:45:55.423 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:45:55.423 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:45:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:45:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:45:55.489 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:55.506 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:45:55.531 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:55.532 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:45:55.532 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:55.602 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-09-17T07:45:55.623 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:55.932 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:56.069 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:56.073 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:56.073 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-17T07:45:56.073 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:56.073 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:45:56.073 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-17T07:45:56.073 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 25 k 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 110 k 2024-09-17T07:45:56.074 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:45:56.180 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:56.209 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:45:56.209 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:56.210 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:45:56.210 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:56.221 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 165 kB/s | 25 kB 00:00 2024-09-17T07:45:56.222 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:56.222 INFO:teuthology.orchestra.run.smithi026.stdout:Total 164 kB/s | 25 kB 00:00 2024-09-17T07:45:56.222 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:45:56.226 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:45:56.226 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:45:56.227 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:45:56.228 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:45:56.311 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:56.311 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 82 k 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 222 k 2024-09-17T07:45:56.312 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:45:56.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-09-17T07:45:56.438 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:56.496 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 449 kB/s | 82 kB 00:00 2024-09-17T07:45:56.496 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:56.496 INFO:teuthology.orchestra.run.smithi094.stdout:Total 446 kB/s | 82 kB 00:00 2024-09-17T07:45:56.496 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:56.503 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:56.503 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:56.564 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:56.564 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:56.833 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:56.991 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:57.054 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:45:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:45:57.054 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:57.080 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:57.123 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:57.124 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:57.124 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:57.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-09-17T07:45:57.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-09-17T07:45:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:57.453 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:57.453 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:57.453 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:45:57.453 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:45:57.453 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:57.453 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:45:57.731 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-09-17T07:45:57.773 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:57.833 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:57.860 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:57.904 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:57.905 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:57.905 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:57.926 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:57.970 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:57.971 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:45:57.972 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:57.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-09-17T07:45:58.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-09-17T07:45:58.315 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:45:58.452 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:45:58.453 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:58.453 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-17T07:45:58.453 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:58.453 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.9 M 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 12 M 2024-09-17T07:45:58.454 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:45:58.552 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:58.639 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:58.640 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:58.683 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:58.684 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:58.684 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:58.732 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:58.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-09-17T07:45:58.776 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:58.778 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:45:58.778 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:58.799 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.5 MB/s | 2.9 MB 00:00 2024-09-17T07:45:58.800 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:45:58.800 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.5 MB/s | 2.9 MB 00:00 2024-09-17T07:45:58.800 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:45:58.809 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:45:58.809 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:45:58.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-09-17T07:45:58.888 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:45:58.889 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:45:59.300 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:45:59.359 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:45:59.375 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:45:59.401 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:45:59.401 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T07:45:59.401 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T07:45:59.401 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T07:45:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:45:59.454 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:59.457 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:45:59.498 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:45:59.499 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:45:59.499 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:45:59.550 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:45:59.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-09-17T07:45:59.595 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:45:59.596 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:45:59.597 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:45:59.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-09-17T07:46:00.174 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:46:00.267 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:00.276 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:00.312 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:46:00.313 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:46:00.313 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:46:00.371 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:00.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-09-17T07:46:00.416 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:00.417 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:46:00.417 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:00.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-09-17T07:46:00.495 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:00.495 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:00.495 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:46:00.495 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:00.495 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:00.495 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:46:00.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-09-17T07:46:00.968 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:46:01.058 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:01.100 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:46:01.101 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:46:01.101 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:46:01.101 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:01.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-09-17T07:46:01.197 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:01.242 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:01.243 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:46:01.243 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:01.307 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:46:01.319 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-09-17T07:46:01.445 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:46:01.446 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:46:01.446 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-17T07:46:01.446 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:46:01.446 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 165 k 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 477 k 2024-09-17T07:46:01.447 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-17T07:46:01.654 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 802 kB/s | 165 kB 00:00 2024-09-17T07:46:01.654 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:01.654 INFO:teuthology.orchestra.run.smithi094.stdout:Total 795 kB/s | 165 kB 00:00 2024-09-17T07:46:01.654 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-17T07:46:01.662 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-17T07:46:01.662 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-17T07:46:01.725 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-17T07:46:01.725 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-17T07:46:01.748 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:46:01.876 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-17T07:46:01.880 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:46:01.880 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:46:01.881 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:01.882 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 82 k 2024-09-17T07:46:01.882 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 222 k 2024-09-17T07:46:01.882 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:46:01.936 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:01.975 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:02.028 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:02.074 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:02.076 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:46:02.076 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:02.081 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 412 kB/s | 82 kB 00:00 2024-09-17T07:46:02.081 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:02.082 INFO:teuthology.orchestra.run.smithi098.stdout:Total 408 kB/s | 82 kB 00:00 2024-09-17T07:46:02.082 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:46:02.089 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:46:02.089 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:46:02.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-09-17T07:46:02.152 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:46:02.152 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:46:02.295 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:46:02.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:02.443 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:02.695 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:02.777 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:02.778 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-17T07:46:02.778 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:02.778 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:02.778 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:46:02.778 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:02.779 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:02.893 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 82 k 2024-09-17T07:46:02.894 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 222 k 2024-09-17T07:46:02.895 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:46:02.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-09-17T07:46:03.068 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:03.068 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:03.068 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:46:03.068 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:03.068 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:03.068 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:46:03.091 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 416 kB/s | 82 kB 00:00 2024-09-17T07:46:03.092 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:03.092 INFO:teuthology.orchestra.run.smithi026.stdout:Total 413 kB/s | 82 kB 00:00 2024-09-17T07:46:03.092 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:46:03.099 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:46:03.100 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:46:03.320 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:46:03.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-09-17T07:46:03.433 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:03.524 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 07:44:20 AM UTC. 2024-09-17T07:46:03.612 INFO:teuthology.orchestra.run.smithi094.stdout:Package rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:03.657 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-17T07:46:03.657 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-17T07:46:03.657 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-17T07:46:03.727 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:03.932 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:46:04.069 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:46:04.070 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-17T07:46:04.070 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:04.070 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.9 M 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 12 M 2024-09-17T07:46:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:46:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:04.101 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:46:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:04.101 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:04.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-09-17T07:46:04.410 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-17T07:46:04.411 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:04.411 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-09-17T07:46:04.411 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:46:04.420 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:46:04.421 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:46:04.499 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:46:04.500 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:46:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:46:04.982 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:05.012 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:46:05.012 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T07:46:05.012 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T07:46:05.012 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T07:46:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:05.124 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:05.126 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-09-17T07:46:05.126 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:05.126 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:46:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-17T07:46:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:05.127 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:46:05.127 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:05.127 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:46:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:05.127 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.9 M 2024-09-17T07:46:05.128 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 12 M 2024-09-17T07:46:05.128 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:46:05.461 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-17T07:46:05.462 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:05.462 INFO:teuthology.orchestra.run.smithi026.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-17T07:46:05.462 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:46:05.472 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:46:05.472 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:46:05.555 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:46:05.556 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:46:05.987 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:46:06.053 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:06.080 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:06.080 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T07:46:06.080 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T07:46:06.080 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T07:46:06.080 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T07:46:06.081 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:06.133 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:06.133 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:46:06.133 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:06.133 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:46:06.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-09-17T07:46:06.971 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:46:07.103 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:46:07.103 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 165 k 2024-09-17T07:46:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 477 k 2024-09-17T07:46:07.105 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-17T07:46:07.246 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:07.247 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:07.247 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:46:07.247 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:07.247 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:07.247 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:07.315 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 785 kB/s | 165 kB 00:00 2024-09-17T07:46:07.315 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:07.316 INFO:teuthology.orchestra.run.smithi098.stdout:Total 779 kB/s | 165 kB 00:00 2024-09-17T07:46:07.316 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-17T07:46:07.323 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-17T07:46:07.323 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-17T07:46:07.387 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-17T07:46:07.387 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-17T07:46:07.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-09-17T07:46:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-17T07:46:07.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:08.093 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:08.111 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:08.248 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-09-17T07:46:08.249 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-09-17T07:46:08.250 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:08.250 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 165 k 2024-09-17T07:46:08.250 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 477 k 2024-09-17T07:46:08.250 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-09-17T07:46:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:08.358 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-17T07:46:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:08.359 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:46:08.464 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 772 kB/s | 165 kB 00:00 2024-09-17T07:46:08.465 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-09-17T07:46:08.465 INFO:teuthology.orchestra.run.smithi026.stdout:Total 767 kB/s | 165 kB 00:00 2024-09-17T07:46:08.465 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-09-17T07:46:08.474 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-09-17T07:46:08.474 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-09-17T07:46:08.541 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-09-17T07:46:08.541 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-09-17T07:46:08.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-09-17T07:46:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-09-17T07:46:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:09.204 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Sep 2024 07:44:19 AM UTC. 2024-09-17T07:46:09.296 INFO:teuthology.orchestra.run.smithi098.stdout:Package rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:09.341 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-17T07:46:09.341 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-17T07:46:09.342 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-17T07:46:09.420 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:09.459 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-17T07:46:09.459 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:09.459 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-09-17T07:46:09.459 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-17T07:46:09.459 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:09.459 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:09.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-09-17T07:46:10.286 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Sep 2024 07:44:16 AM UTC. 2024-09-17T07:46:10.379 INFO:teuthology.orchestra.run.smithi026.stdout:Package rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-17T07:46:10.425 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-09-17T07:46:10.426 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-09-17T07:46:10.426 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-09-17T07:46:10.516 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:10.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:46:10.663 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T07:46:10.685 INFO:teuthology.orchestra.run.smithi026.stdout:17.2.7-1643.gd75f2382.el9 2024-09-17T07:46:10.686 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-17T07:46:10.686 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-17T07:46:10.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:46:10.827 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T07:46:10.853 INFO:teuthology.orchestra.run.smithi094.stdout:17.2.7-1643.gd75f2382.el9 2024-09-17T07:46:10.853 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-17T07:46:10.853 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-17T07:46:10.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:46:10.996 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T07:46:11.021 INFO:teuthology.orchestra.run.smithi098.stdout:17.2.7-1643.gd75f2382.el9 2024-09-17T07:46:11.021 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-17T07:46:11.022 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-17T07:46:11.024 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T07:46:11.050 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:11.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T07:46:11.079 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:11.079 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T07:46:11.110 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:11.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T07:46:11.140 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T07:46:11.140 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:11.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T07:46:11.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T07:46:11.233 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:11.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T07:46:11.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T07:46:11.331 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:11.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T07:46:11.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T07:46:11.428 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T07:46:11.429 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:11.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T07:46:11.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T07:46:11.522 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:11.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T07:46:11.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T07:46:11.620 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:11.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T07:46:11.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T07:46:11.717 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T07:46:11.718 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:11.718 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T07:46:11.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T07:46:11.810 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:11.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T07:46:11.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T07:46:11.906 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:11.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T07:46:11.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T07:46:12.001 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T07:46:12.428 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T07:46:12.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-09-17T07:46:12.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-09-17T07:46:12.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-09-17T07:46:12.463 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T07:46:12.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T07:46:12.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T07:46:12.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T07:46:12.533 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T07:46:12.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T07:46:12.564 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T07:46:12.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T07:46:12.610 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T07:46:12.610 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T07:46:12.610 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_latest} thrashers/cache thrashosds-health workloads/rbd_nbd}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7908725', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbxAaFhqneXjYqjKjSehs56lFBY4Uc6xiViCJANlTguvg/3ChXysKLOyg2HcMsvzCdtNGLTXPt7Dmop8hza4Do=', 'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHtzmXtZHbwaGAlfGXCePRZVRFqtsvky2Uh/KaRUznrd0dwRXNUcoLDxweo248bA0G7Bq2OXNjBxXqziFyfE3k=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCo8LzbKvqDykYK0VR9mxHzPGyEBOkkYVfQHoXx0g/cifqhd0vc6EpK3+jcuA1Uafw7jMt/oh8ki/6h85YlyyUg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_07:28:46', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T07:46:12.611 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T07:46:12.630 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T07:46:12.640 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T07:46:12.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T07:46:12.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T07:46:12.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T07:46:12.726 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:12.726 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T07:46:12.766 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T07:46:12.766 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-09-17T07:46:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-17 07:46:06.324016787 +0000 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-17 07:43:38.295285399 +0000 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-17 07:43:38.295285399 +0000 2024-09-17T07:46:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-17 07:43:38.295285399 +0000 2024-09-17T07:46:12.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T07:46:12.887 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-17T07:46:12.887 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-17T07:46:12.887 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000101001 s, 5.1 MB/s 2024-09-17T07:46:12.888 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T07:46:12.945 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-09-17T07:46:13.001 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T07:46:13.001 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:13.001 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T07:46:13.001 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:13.002 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:13.002 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-17 07:46:06.324016787 +0000 2024-09-17T07:46:13.002 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-17 07:43:38.628280480 +0000 2024-09-17T07:46:13.002 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-17 07:43:38.628280480 +0000 2024-09-17T07:46:13.002 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-17 07:43:38.628280480 +0000 2024-09-17T07:46:13.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T07:46:13.065 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-17T07:46:13.066 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-17T07:46:13.066 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.00010594 s, 4.8 MB/s 2024-09-17T07:46:13.067 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T07:46:13.124 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-09-17T07:46:13.180 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T07:46:13.180 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:13.180 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-17T07:46:13.181 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:13.181 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:13.181 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-17 07:46:06.325016771 +0000 2024-09-17T07:46:13.181 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-17 07:43:38.970275428 +0000 2024-09-17T07:46:13.181 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-17 07:43:38.970275428 +0000 2024-09-17T07:46:13.181 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-17 07:43:38.970275428 +0000 2024-09-17T07:46:13.181 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T07:46:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-17T07:46:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-17T07:46:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.9152e-05 s, 5.2 MB/s 2024-09-17T07:46:13.247 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T07:46:13.304 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-09-17T07:46:13.360 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T07:46:13.360 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:13.360 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-17T07:46:13.360 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:13.361 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:13.361 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-17 07:46:06.325016771 +0000 2024-09-17T07:46:13.361 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-17 07:43:39.303270509 +0000 2024-09-17T07:46:13.361 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-17 07:43:39.303270509 +0000 2024-09-17T07:46:13.361 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-09-17 07:43:39.303270509 +0000 2024-09-17T07:46:13.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T07:46:13.424 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-17T07:46:13.424 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-17T07:46:13.424 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.6569e-05 s, 5.3 MB/s 2024-09-17T07:46:13.425 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T07:46:13.482 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:13.482 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:13.482 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T07:46:13.501 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T07:46:13.502 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-09-17T07:46:13.559 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T07:46:13.559 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:13.559 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-17T07:46:13.560 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:13.560 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:13.560 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-17 07:45:59.663771666 +0000 2024-09-17T07:46:13.560 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-17 07:43:33.392911783 +0000 2024-09-17T07:46:13.560 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-17 07:43:33.392911783 +0000 2024-09-17T07:46:13.560 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-17 07:43:33.392911783 +0000 2024-09-17T07:46:13.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T07:46:13.625 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-17T07:46:13.626 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-17T07:46:13.626 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000118352 s, 4.3 MB/s 2024-09-17T07:46:13.627 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T07:46:13.685 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-09-17T07:46:13.743 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-17 07:45:59.663771666 +0000 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-17 07:43:33.695906694 +0000 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-17 07:43:33.695906694 +0000 2024-09-17T07:46:13.744 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-17 07:43:33.695906694 +0000 2024-09-17T07:46:13.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T07:46:13.808 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-17T07:46:13.808 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-17T07:46:13.808 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000118604 s, 4.3 MB/s 2024-09-17T07:46:13.809 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T07:46:13.866 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-17 07:45:59.663771666 +0000 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-17 07:43:34.018901267 +0000 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-17 07:43:34.018901267 +0000 2024-09-17T07:46:13.923 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-17 07:43:34.018901267 +0000 2024-09-17T07:46:13.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T07:46:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-17T07:46:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-17T07:46:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000108433 s, 4.7 MB/s 2024-09-17T07:46:13.989 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T07:46:14.046 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-09-17T07:46:14.102 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T07:46:14.102 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-17 07:45:59.664771649 +0000 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-17 07:43:34.329896041 +0000 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-17 07:43:34.329896041 +0000 2024-09-17T07:46:14.103 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-17 07:43:34.329896041 +0000 2024-09-17T07:46:14.103 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T07:46:14.166 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-17T07:46:14.166 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-17T07:46:14.167 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000126622 s, 4.0 MB/s 2024-09-17T07:46:14.168 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T07:46:14.225 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:14.225 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T07:46:14.225 INFO:tasks.ceph:Generating config... 2024-09-17T07:46:14.249 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T07:46:14.249 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-17T07:46:14.249 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-17T07:46:14.249 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T07:46:14.249 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T07:46:14.249 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T07:46:14.249 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T07:46:14.249 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T07:46:14.250 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T07:46:14.251 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T07:46:14.251 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-17T07:46:14.251 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T07:46:14.251 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T07:46:14.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T07:46:14.299 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T07:46:14.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T07:46:14.346 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T07:46:14.413 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.26'), ('mon.c', '[v2:172.21.15.26:3301,v1:172.21.15.26:6790]'), ('mon.b', '172.21.15.94')] 2024-09-17T07:46:14.413 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.26,[v2:172.21.15.26:3301,v1:172.21.15.26:6790],172.21.15.94', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-17T07:46:14.414 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:14.414 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T07:46:14.470 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.26 --addv c '[v2:172.21.15.26:3301,v1:172.21.15.26:6790]' --add b 172.21.15.94 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:14.554 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 666af130-c9e0-433a-95ce-0b7c815e9e68 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = octopus 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 666af130-c9e0-433a-95ce-0b7c815e9e68 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2024-09-17T07:46:14.554988+0000 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-09-17T07:46:14.554988+0000 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 15 (octopus) 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 1 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.b 2024-09-17T07:46:14.555 INFO:teuthology.orchestra.run.smithi026.stdout:2: [v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0] mon.c 2024-09-17T07:46:14.556 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T07:46:14.557 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T07:46:14.573 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 666af130-c9e0-433a-95ce-0b7c815e9e68... 2024-09-17T07:46:14.574 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T07:46:14.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T07:46:14.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T07:46:14.658 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-09-17T07:46:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-09-17T07:46:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T07:46:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-09-17T07:46:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-09-17T07:46:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-09-17T07:46:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-09-17T07:46:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-09-17T07:46:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-09-17T07:46:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-09-17T07:46:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T07:46:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-09-17T07:46:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T07:46:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T07:46:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-09-17T07:46:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-09-17T07:46:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-09-17T07:46:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-09-17T07:46:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-09-17T07:46:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-09-17T07:46:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-09-17T07:46:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-09-17T07:46:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T07:46:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T07:46:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-09-17T07:46:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T07:46:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-09-17T07:46:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-09-17T07:46:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-09-17T07:46:14.667 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-09-17T07:46:14.667 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-09-17T07:46:14.667 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "172.21.15.26,[v2:172.21.15.26:3301,v1:172.21.15.26:6790],172.21.15.94" 2024-09-17T07:46:14.667 INFO:teuthology.orchestra.run.smithi026.stdout: mon client directed command retry = 5 2024-09-17T07:46:14.667 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on pool no app = False 2024-09-17T07:46:14.668 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject socket failures = 5000 2024-09-17T07:46:14.668 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 666af130-c9e0-433a-95ce-0b7c815e9e68 2024-09-17T07:46:14.668 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.668 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-09-17T07:46:14.668 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-09-17T07:46:14.668 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.669 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-09-17T07:46:14.669 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-09-17T07:46:14.669 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-09-17T07:46:14.669 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.669 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-09-17T07:46:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-09-17T07:46:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-09-17T07:46:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-09-17T07:46:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-09-17T07:46:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-09-17T07:46:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-09-17T07:46:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-09-17T07:46:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-09-17T07:46:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-09-17T07:46:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-09-17T07:46:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-09-17T07:46:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-09-17T07:46:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-09-17T07:46:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: bluefs allocator = hybrid 2024-09-17T07:46:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore allocator = hybrid 2024-09-17T07:46:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2024-09-17T07:46:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2024-09-17T07:46:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 1/20 2024-09-17T07:46:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 1/20 2024-09-17T07:46:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-17T07:46:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-09-17T07:46:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 4/10 2024-09-17T07:46:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T07:46:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2024-09-17T07:46:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2024-09-17T07:46:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2024-09-17T07:46:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: osd max markdown count = 1000 2024-09-17T07:46:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2024-09-17T07:46:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.676 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-09-17T07:46:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-17T07:46:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-09-17T07:46:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-09-17T07:46:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-09-17T07:46:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-09-17T07:46:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-09-17T07:46:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-09-17T07:46:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.678 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-09-17T07:46:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-17T07:46:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-09-17T07:46:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-09-17T07:46:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-09-17T07:46:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-09-17T07:46:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-09-17T07:46:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-09-17T07:46:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-09-17T07:46:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-09-17T07:46:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-09-17T07:46:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T07:46:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T07:46:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T07:46:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T07:46:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T07:46:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T07:46:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-09-17T07:46:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-09-17T07:46:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-09-17T07:46:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:14.683 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-09-17T07:46:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-09-17T07:46:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-09-17T07:46:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-09-17T07:46:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T07:46:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T07:46:14.684 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-09-17T07:46:14.684 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2024-09-17T07:46:14.684 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2024-09-17T07:46:14.688 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-17T07:46:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-09-17T07:46:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T07:46:14.689 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-09-17T07:46:14.689 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.689 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-09-17T07:46:14.689 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.689 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-09-17T07:46:14.690 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.690 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-09-17T07:46:14.690 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-09-17T07:46:14.690 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.690 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-09-17T07:46:14.690 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-09-17T07:46:14.691 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.691 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T07:46:14.691 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-09-17T07:46:14.691 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.691 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T07:46:14.692 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T07:46:14.692 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.692 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-09-17T07:46:14.692 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.692 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T07:46:14.693 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T07:46:14.694 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-09-17T07:46:14.694 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.694 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T07:46:14.694 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.694 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-09-17T07:46:14.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.696 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-09-17T07:46:14.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.696 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-09-17T07:46:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-09-17T07:46:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-09-17T07:46:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.26,[v2:172.21.15.26:3301,v1:172.21.15.26:6790],172.21.15.94" 2024-09-17T07:46:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-09-17T07:46:14.697 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-09-17T07:46:14.698 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-09-17T07:46:14.698 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 666af130-c9e0-433a-95ce-0b7c815e9e68 2024-09-17T07:46:14.698 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.698 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-09-17T07:46:14.698 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-09-17T07:46:14.698 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.699 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-09-17T07:46:14.699 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-09-17T07:46:14.699 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-09-17T07:46:14.699 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.699 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-09-17T07:46:14.700 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-09-17T07:46:14.700 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.700 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-09-17T07:46:14.700 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-09-17T07:46:14.700 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-09-17T07:46:14.701 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.701 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-09-17T07:46:14.701 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-09-17T07:46:14.701 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.701 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-09-17T07:46:14.702 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.702 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-09-17T07:46:14.702 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.702 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-09-17T07:46:14.702 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.702 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-09-17T07:46:14.703 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-09-17T07:46:14.703 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.703 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-09-17T07:46:14.703 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-09-17T07:46:14.703 INFO:teuthology.orchestra.run.smithi098.stdout: bluefs allocator = hybrid 2024-09-17T07:46:14.703 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = hybrid 2024-09-17T07:46:14.704 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-09-17T07:46:14.704 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-09-17T07:46:14.704 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1/20 2024-09-17T07:46:14.704 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1/20 2024-09-17T07:46:14.704 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-17T07:46:14.704 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-09-17T07:46:14.705 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 4/10 2024-09-17T07:46:14.705 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T07:46:14.705 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-09-17T07:46:14.705 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-09-17T07:46:14.705 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: osd max markdown count = 1000 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.706 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T07:46:14.707 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-09-17T07:46:14.708 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-09-17T07:46:14.709 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T07:46:14.710 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.711 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "172.21.15.26,[v2:172.21.15.26:3301,v1:172.21.15.26:6790],172.21.15.94" 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: mon client directed command retry = 5 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on pool no app = False 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject socket failures = 5000 2024-09-17T07:46:14.712 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 666af130-c9e0-433a-95ce-0b7c815e9e68 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-09-17T07:46:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: bluefs allocator = hybrid 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore allocator = hybrid 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 1/20 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 1/20 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 4/10 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-09-17T07:46:14.715 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: osd max markdown count = 1000 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.716 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T07:46:14.717 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-09-17T07:46:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-09-17T07:46:14.719 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T07:46:14.719 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T07:46:14.719 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-09-17T07:46:14.719 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2024-09-17T07:46:14.719 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2024-09-17T07:46:14.719 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T07:46:14.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T07:46:14.779 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T07:46:14.780 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:14.780 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T07:46:14.795 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:14.796 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T07:46:14.851 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:46:14.851 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:14.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T07:46:14.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T07:46:14.927 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:14.927 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:14.983 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:46:14.984 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:14.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T07:46:14.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T07:46:15.020 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:15.020 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:15.076 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-09-17T07:46:15.077 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:15.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T07:46:15.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T07:46:15.117 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:15.117 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:15.175 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T07:46:15.175 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T07:46:15.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T07:46:15.231 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T07:46:15.234 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T07:46:15.234 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T07:46:15.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T07:46:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T07:46:15.301 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T07:46:15.302 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T07:46:15.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T07:46:15.328 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:15.328 INFO:tasks.ceph:role: osd.0 2024-09-17T07:46:15.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:46:15.329 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T07:46:15.393 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T07:46:15.394 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T07:46:15.426 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-17T07:46:15.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-17T07:46:15.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T07:46:15.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T07:46:15.570 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T07:46:15.637 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:15.637 INFO:tasks.ceph:role: osd.1 2024-09-17T07:46:15.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:46:15.637 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-17T07:46:15.703 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T07:46:15.704 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T07:46:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T07:46:15.704 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T07:46:15.736 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-17T07:46:15.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-17T07:46:15.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T07:46:15.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T07:46:15.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T07:46:15.946 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:15.946 INFO:tasks.ceph:role: osd.2 2024-09-17T07:46:15.946 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:46:15.947 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T07:46:16.012 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T07:46:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T07:46:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T07:46:16.013 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T07:46:16.044 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-17T07:46:16.047 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-17T07:46:16.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T07:46:16.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T07:46:16.198 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:16.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.290+0000 7eff3b246440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T07:46:16.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.290+0000 7eff3b246440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T07:46:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.290+0000 7eff3b246440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-17T07:46:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.290+0000 7eff3b246440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-17T07:46:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.290+0000 7eff3b246440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-17T07:46:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.291+0000 7eff3b246440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T07:46:16.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:16.815+0000 7eff3b246440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:17.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:17.579+0000 7eff3b246440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:18.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:18.343+0000 7eff3b246440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:19.141 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T07:46:19.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:19.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.258+0000 7fc04ad23440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T07:46:19.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.258+0000 7fc04ad23440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T07:46:19.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.258+0000 7fc04ad23440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-17T07:46:19.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.258+0000 7fc04ad23440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-17T07:46:19.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.258+0000 7fc04ad23440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-17T07:46:19.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.259+0000 7fc04ad23440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T07:46:19.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:19.790+0000 7fc04ad23440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:20.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:20.556+0000 7fc04ad23440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:21.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:21.317+0000 7fc04ad23440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:22.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:46:22.149 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.240+0000 7f5b4368b440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T07:46:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.240+0000 7f5b4368b440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T07:46:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.240+0000 7f5b4368b440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-17T07:46:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.240+0000 7f5b4368b440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-17T07:46:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.240+0000 7f5b4368b440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-17T07:46:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.241+0000 7f5b4368b440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T07:46:22.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:22.775+0000 7f5b4368b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:23.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:23.543+0000 7f5b4368b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:24.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:46:24.306+0000 7f5b4368b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:25.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T07:46:25.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T07:46:25.168 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:25.169 INFO:tasks.ceph:role: osd.3 2024-09-17T07:46:25.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:46:25.169 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T07:46:25.236 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T07:46:25.237 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T07:46:25.269 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-09-17T07:46:25.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-09-17T07:46:25.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-17T07:46:25.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T07:46:25.412 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T07:46:25.478 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:25.478 INFO:tasks.ceph:role: osd.4 2024-09-17T07:46:25.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:46:25.478 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T07:46:25.543 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T07:46:25.544 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T07:46:25.544 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T07:46:25.544 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T07:46:25.544 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T07:46:25.544 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-09-17T07:46:25.544 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T07:46:25.545 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T07:46:25.545 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T07:46:25.545 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T07:46:25.577 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-09-17T07:46:25.579 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-09-17T07:46:25.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-17T07:46:25.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T07:46:25.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T07:46:25.785 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T07:46:25.786 INFO:tasks.ceph:role: osd.5 2024-09-17T07:46:25.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:46:25.786 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T07:46:25.850 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T07:46:25.851 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T07:46:25.883 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-09-17T07:46:25.885 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-09-17T07:46:25.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-17T07:46:25.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T07:46:26.035 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:26.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.143+0000 7ffa2af35440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T07:46:26.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.143+0000 7ffa2af35440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T07:46:26.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.143+0000 7ffa2af35440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-17T07:46:26.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.143+0000 7ffa2af35440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-17T07:46:26.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.143+0000 7ffa2af35440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-17T07:46:26.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.143+0000 7ffa2af35440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T07:46:26.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:26.678+0000 7ffa2af35440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:27.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:27.447+0000 7ffa2af35440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:28.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:28.210+0000 7ffa2af35440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:29.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T07:46:29.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:29.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.123+0000 7f0908127440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T07:46:29.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.124+0000 7f0908127440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T07:46:29.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.124+0000 7f0908127440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-17T07:46:29.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.124+0000 7f0908127440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-17T07:46:29.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.124+0000 7f0908127440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-17T07:46:29.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.124+0000 7f0908127440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T07:46:29.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:29.648+0000 7f0908127440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:30.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:30.409+0000 7f0908127440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:31.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:31.172+0000 7f0908127440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:31.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T07:46:31.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:32.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.079+0000 7f785944e440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T07:46:32.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.079+0000 7f785944e440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T07:46:32.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.079+0000 7f785944e440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-17T07:46:32.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.079+0000 7f785944e440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-17T07:46:32.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.079+0000 7f785944e440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-17T07:46:32.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.079+0000 7f785944e440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T07:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:32.602+0000 7f785944e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:33.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:33.369+0000 7f785944e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:46:34.131+0000 7f785944e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-17T07:46:34.928 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:46:34.957 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T07:46:34.959 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:34.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T07:46:34.991 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:34.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T07:46:35.061 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:35.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T07:46:35.130 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:35.130 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T07:46:35.160 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:35.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T07:46:35.229 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:35.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T07:46:35.297 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:35.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T07:46:35.365 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:46:35.365 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T07:46:35.384 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T07:46:35.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T07:46:35.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T07:46:35.414 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-09-17T07:46:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDIM+lmG0ttERAA1BA/MqMqGPnlrCAIeRQ/bg== 2024-09-17T07:46:35.414 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-09-17T07:46:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDLM+lmV+54DxAAk8Z5U3hgfkl12U7VaYulDw== 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDOM+lmYJFiDhAACD0do7HFSBV56sjlKifa3g== 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDHM+lmn13VDRAAv+kg1cfRUL6kvDGtRPCLdQ== 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDSM+lmvbqUCBAARZ1mR88eCbEclMo4VwyE3w== 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDVM+lmfjFtBxAA+3cjd1E2lpqVNJtY/0COgg== 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDYM+lmKE7CBBAAiV8fMHqgWYkLpgHwrxbVsA== 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-09-17T07:46:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDHM+lmkyMvERAANuWtqLC4ytfEV0vAIMppwA== 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDIM+lmG0ttERAA1BA/MqMqGPnlrCAIeRQ/bg== 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDLM+lmV+54DxAAk8Z5U3hgfkl12U7VaYulDw== 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDOM+lmYJFiDhAACD0do7HFSBV56sjlKifa3g== 2024-09-17T07:46:35.430 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDHM+lmn13VDRAAv+kg1cfRUL6kvDGtRPCLdQ== 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDSM+lmvbqUCBAARZ1mR88eCbEclMo4VwyE3w== 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDVM+lmfjFtBxAA+3cjd1E2lpqVNJtY/0COgg== 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDYM+lmKE7CBBAAiV8fMHqgWYkLpgHwrxbVsA== 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-09-17T07:46:35.431 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDHM+lmkyMvERAANuWtqLC4ytfEV0vAIMppwA== 2024-09-17T07:46:35.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.459 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T07:46:35.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T07:46:35.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T07:46:35.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T07:46:35.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T07:46:35.865 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T07:46:35.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T07:46:35.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T07:46:35.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T07:46:36.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T07:46:36.076 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T07:46:36.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T07:46:36.194 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T07:46:36.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T07:46:36.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T07:46:36.337 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:36.340 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T07:46:36.392 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T07:46:36.396 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T07:46:36.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T07:46:36.398 INFO:tasks.ceph.mon.a:Started 2024-09-17T07:46:36.399 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T07:46:36.399 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T07:46:36.401 INFO:tasks.ceph.mon.c:Started 2024-09-17T07:46:36.401 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T07:46:36.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T07:46:36.435 INFO:tasks.ceph.mon.b:Started 2024-09-17T07:46:36.435 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T07:46:36.435 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T07:46:36.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T07:46:36.437 INFO:tasks.ceph.mgr.x:Started 2024-09-17T07:46:36.437 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T07:46:36.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T07:46:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-09-17T07:46:41.797 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T07:46:41.797 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:41.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T07:46:41.817 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:41.816+0000 7fd5547c9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:46:41.824 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:41.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T07:46:41.856 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:41.856+0000 7fd5547c9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:46:41.890 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-17T07:46:41.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T07:46:41.905 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:41.905+0000 7fd5547c9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:46:41.956 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:41.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T07:46:41.982 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:41.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T07:46:42.047 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-17T07:46:42.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T07:46:42.059 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:42.058+0000 7fd5547c9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:46:42.083 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1 0 2024-09-17T07:46:42.171 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:42.171+0000 7fd5547c9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:46:42.303 INFO:teuthology.orchestra.run.smithi094.stdout:0 2024-09-17T07:46:42.314 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new b26da222-1990-4e5d-a449-0ff9d561090f 1 2024-09-17T07:46:42.470 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:42.469+0000 7fd5547c9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:46:42.486 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-09-17T07:46:42.497 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new e6761959-1933-4d0a-a0ff-b437b983611e 2 2024-09-17T07:46:42.530 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:42.529+0000 7fd5547c9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:46:42.668 INFO:teuthology.orchestra.run.smithi094.stdout:2 2024-09-17T07:46:42.669 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:42.668+0000 7fd5547c9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:46:42.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 5166dbfd-44ec-4d31-b486-b151dfde1443 3 2024-09-17T07:46:42.889 INFO:teuthology.orchestra.run.smithi094.stdout:3 2024-09-17T07:46:42.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new f7829bfc-4efc-4bc9-8c4d-423690de31fd 4 2024-09-17T07:46:43.071 INFO:teuthology.orchestra.run.smithi094.stdout:4 2024-09-17T07:46:43.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 4e0679f8-a785-4804-93d3-79b7acabe009 5 2024-09-17T07:46:43.253 INFO:teuthology.orchestra.run.smithi094.stdout:5 2024-09-17T07:46:43.264 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T07:46:43.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T07:46:43.267 INFO:tasks.ceph.osd.0:Started 2024-09-17T07:46:43.267 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T07:46:43.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T07:46:43.269 INFO:tasks.ceph.osd.1:Started 2024-09-17T07:46:43.270 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T07:46:43.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T07:46:43.272 INFO:tasks.ceph.osd.2:Started 2024-09-17T07:46:43.272 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T07:46:43.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T07:46:43.274 INFO:tasks.ceph.osd.3:Started 2024-09-17T07:46:43.274 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T07:46:43.275 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T07:46:43.276 INFO:tasks.ceph.osd.4:Started 2024-09-17T07:46:43.277 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T07:46:43.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T07:46:43.278 INFO:tasks.ceph.osd.5:Started 2024-09-17T07:46:43.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:46:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:43.459 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":8,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:46:43.251109+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:46:43.470 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T07:46:43.470 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T07:46:43.643 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:43.642+0000 7fd5547c9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:46:43.692 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:43.692+0000 7fd5547c9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:46:43.870 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:46:43.869+0000 7fbfce43b440 -1 Falling back to public interface 2024-09-17T07:46:43.874 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:46:43.873+0000 7fd820e0d440 -1 Falling back to public interface 2024-09-17T07:46:43.875 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:46:43.874+0000 7f500ae89440 -1 Falling back to public interface 2024-09-17T07:46:43.876 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:46:43.876+0000 7fd81edfd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6819/0: (98) Address already in use 2024-09-17T07:46:43.878 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:46:43.877+0000 7f488e70f440 -1 Falling back to public interface 2024-09-17T07:46:43.879 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:46:43.879+0000 7f990b553440 -1 Falling back to public interface 2024-09-17T07:46:43.882 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:46:43.882+0000 7f168dc53440 -1 Falling back to public interface 2024-09-17T07:46:44.055 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.055+0000 7fd5547c9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:46:44.074 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T07:46:44.136 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.135+0000 7fd5547c9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:46:44.225 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.225+0000 7fd5547c9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:46:44.231 INFO:teuthology.misc.health.smithi026.stdout: 2024-09-17T07:46:44.231 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":8,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:46:43.251109+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:46:44.240 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-17T07:46:44.269 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.268+0000 7fd5547c9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:46:44.361 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.361+0000 7fd5547c9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:46:44.412 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.412+0000 7fd5547c9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:46:44.457 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.457+0000 7fd5547c9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:46:44.548 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.548+0000 7fd5547c9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:46:44.596 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.595+0000 7fd5547c9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:46:44.644 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.644+0000 7fd5547c9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:46:44.710 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.710+0000 7fd5547c9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:46:44.754 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.753+0000 7fd5547c9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:46:44.795 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:44.795+0000 7fd5547c9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:46:45.401 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:45.401+0000 7fd5547c9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:46:46.415 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:46.413+0000 7fd538e49640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T07:46:46.415 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-09-17T07:46:46.413+0000 7fd538e49640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T07:46:47.319 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:46:47.318+0000 7fbfce43b440 -1 osd.0 0 log_to_monitors true 2024-09-17T07:46:47.361 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:46:47.360+0000 7fd820e0d440 -1 osd.1 0 log_to_monitors true 2024-09-17T07:46:47.364 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:46:47.363+0000 7f990b553440 -1 osd.4 0 log_to_monitors true 2024-09-17T07:46:47.394 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:46:47.393+0000 7f500ae89440 -1 osd.2 0 log_to_monitors true 2024-09-17T07:46:47.409 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:46:47.407+0000 7f168dc53440 -1 osd.5 0 log_to_monitors true 2024-09-17T07:46:47.411 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:46:47.410+0000 7f488e70f440 -1 osd.3 0 log_to_monitors true 2024-09-17T07:46:49.981 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:46:49.979+0000 7f16883f6640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T07:46:49.982 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:46:49.980+0000 7f99077f9640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T07:46:49.986 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:46:49.985+0000 7f16813e8640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:46:49.987 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:46:49.986+0000 7f98fefe8640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:46:49.989 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:46:49.987+0000 7f48897f6640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T07:46:49.994 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:46:49.992+0000 7f48827e8640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:46:50.009 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:46:50.008+0000 7f50077f9640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T07:46:50.016 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:46:50.015+0000 7f4ffee84640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:46:50.036 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:46:50.036+0000 7fd81b5f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T07:46:50.042 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:46:50.042+0000 7fd8145e8640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:46:50.045 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:46:50.045+0000 7fbfcadf9640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T07:46:50.051 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:46:50.050+0000 7fbfc2484640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:46:50.846 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T07:46:51.127 INFO:teuthology.misc.health.smithi026.stdout: 2024-09-17T07:46:51.127 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:46:50.986897+0000","last_up_change":"2024-09-17T07:46:50.986897+0000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":31884},{"type":"v1","addr":"172.21.15.26:6801","nonce":31884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":31884},{"type":"v1","addr":"172.21.15.26:6803","nonce":31884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":31884},{"type":"v1","addr":"172.21.15.26:6807","nonce":31884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":31884},{"type":"v1","addr":"172.21.15.26:6805","nonce":31884}]},"public_addr":"172.21.15.26:6801/31884","cluster_addr":"172.21.15.26:6803/31884","heartbeat_back_addr":"172.21.15.26:6807/31884","heartbeat_front_addr":"172.21.15.26:6805/31884","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":31604},{"type":"v1","addr":"172.21.15.94:6801","nonce":31604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":31604},{"type":"v1","addr":"172.21.15.94:6803","nonce":31604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":31604},{"type":"v1","addr":"172.21.15.94:6807","nonce":31604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":31604},{"type":"v1","addr":"172.21.15.94:6805","nonce":31604}]},"public_addr":"172.21.15.94:6801/31604","cluster_addr":"172.21.15.94:6803/31604","heartbeat_back_addr":"172.21.15.94:6807/31604","heartbeat_front_addr":"172.21.15.94:6805/31604","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":31606},{"type":"v1","addr":"172.21.15.94:6817","nonce":31606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":31606},{"type":"v1","addr":"172.21.15.94:6819","nonce":31606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":31606},{"type":"v1","addr":"172.21.15.94:6823","nonce":31606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":31606},{"type":"v1","addr":"172.21.15.94:6821","nonce":31606}]},"public_addr":"172.21.15.94:6817/31606","cluster_addr":"172.21.15.94:6819/31606","heartbeat_back_addr":"172.21.15.94:6823/31606","heartbeat_front_addr":"172.21.15.94:6821/31606","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:46:51.138 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-17T07:46:51.138 INFO:tasks.ceph:Creating RBD pool 2024-09-17T07:46:51.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-17T07:46:51.992 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2024-09-17T07:46:52.007 DEBUG:teuthology.orchestra.run.smithi026:> rbd --cluster ceph pool init rbd 2024-09-17T07:46:55.009 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T07:46:55.010 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T07:46:55.010 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T07:46:55.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T07:46:55.010 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T07:46:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:55.350 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":31179},{"type":"v1","addr":"172.21.15.94:6825","nonce":31179}]},"active_addr":"172.21.15.94:6825/31179","active_change":"2024-09-17T07:46:45.403704+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3945451697}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3828011955}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3044391845}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1837279647}]}]} 2024-09-17T07:46:55.352 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T07:46:55.352 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T07:46:55.352 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:46:55.625 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:55.625 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:46:54.996650+0000","last_up_change":"2024-09-17T07:46:50.986897+0000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":31884},{"type":"v1","addr":"172.21.15.26:6801","nonce":31884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":31884},{"type":"v1","addr":"172.21.15.26:6803","nonce":31884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":31884},{"type":"v1","addr":"172.21.15.26:6807","nonce":31884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":31884},{"type":"v1","addr":"172.21.15.26:6805","nonce":31884}]},"public_addr":"172.21.15.26:6801/31884","cluster_addr":"172.21.15.26:6803/31884","heartbeat_back_addr":"172.21.15.26:6807/31884","heartbeat_front_addr":"172.21.15.26:6805/31884","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":31604},{"type":"v1","addr":"172.21.15.94:6801","nonce":31604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":31604},{"type":"v1","addr":"172.21.15.94:6803","nonce":31604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":31604},{"type":"v1","addr":"172.21.15.94:6807","nonce":31604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":31604},{"type":"v1","addr":"172.21.15.94:6805","nonce":31604}]},"public_addr":"172.21.15.94:6801/31604","cluster_addr":"172.21.15.94:6803/31604","heartbeat_back_addr":"172.21.15.94:6807/31604","heartbeat_front_addr":"172.21.15.94:6805/31604","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":31606},{"type":"v1","addr":"172.21.15.94:6817","nonce":31606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":31606},{"type":"v1","addr":"172.21.15.94:6819","nonce":31606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":31606},{"type":"v1","addr":"172.21.15.94:6823","nonce":31606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":31606},{"type":"v1","addr":"172.21.15.94:6821","nonce":31606}]},"public_addr":"172.21.15.94:6817/31606","cluster_addr":"172.21.15.94:6819/31606","heartbeat_back_addr":"172.21.15.94:6823/31606","heartbeat_front_addr":"172.21.15.94:6821/31606","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:46:55.637 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T07:46:55.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:46:55.908 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:55.908 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:46:54.996650+0000","last_up_change":"2024-09-17T07:46:50.986897+0000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":31884},{"type":"v1","addr":"172.21.15.26:6801","nonce":31884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":31884},{"type":"v1","addr":"172.21.15.26:6803","nonce":31884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":31884},{"type":"v1","addr":"172.21.15.26:6807","nonce":31884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":31884},{"type":"v1","addr":"172.21.15.26:6805","nonce":31884}]},"public_addr":"172.21.15.26:6801/31884","cluster_addr":"172.21.15.26:6803/31884","heartbeat_back_addr":"172.21.15.26:6807/31884","heartbeat_front_addr":"172.21.15.26:6805/31884","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":31604},{"type":"v1","addr":"172.21.15.94:6801","nonce":31604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":31604},{"type":"v1","addr":"172.21.15.94:6803","nonce":31604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":31604},{"type":"v1","addr":"172.21.15.94:6807","nonce":31604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":31604},{"type":"v1","addr":"172.21.15.94:6805","nonce":31604}]},"public_addr":"172.21.15.94:6801/31604","cluster_addr":"172.21.15.94:6803/31604","heartbeat_back_addr":"172.21.15.94:6807/31604","heartbeat_front_addr":"172.21.15.94:6805/31604","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":31606},{"type":"v1","addr":"172.21.15.94:6817","nonce":31606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":31606},{"type":"v1","addr":"172.21.15.94:6819","nonce":31606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":31606},{"type":"v1","addr":"172.21.15.94:6823","nonce":31606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":31606},{"type":"v1","addr":"172.21.15.94:6821","nonce":31606}]},"public_addr":"172.21.15.94:6817/31606","cluster_addr":"172.21.15.94:6819/31606","heartbeat_back_addr":"172.21.15.94:6823/31606","heartbeat_front_addr":"172.21.15.94:6821/31606","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:46:55.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T07:46:55.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T07:46:55.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T07:46:55.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T07:46:55.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T07:46:55.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T07:46:56.092 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-09-17T07:46:56.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T07:46:56.094 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:56.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T07:46:56.121 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:56.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T07:46:56.125 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:56.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T07:46:56.132 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:56.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T07:46:56.134 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:56.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T07:46:56.437 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:56.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.1 2024-09-17T07:46:56.497 INFO:teuthology.orchestra.run.smithi026.stdout:47244640257 2024-09-17T07:46:56.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.3 2024-09-17T07:46:56.538 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:56.553 INFO:teuthology.orchestra.run.smithi026.stdout:47244640257 2024-09-17T07:46:56.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-17T07:46:56.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.0 2024-09-17T07:46:56.585 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:56.585 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:56.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-17T07:46:56.600 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-17T07:46:57.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T07:46:57.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T07:46:57.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T07:46:57.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T07:46:57.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T07:46:57.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T07:46:57.829 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:57.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.1 2024-09-17T07:46:57.879 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:57.895 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-17T07:46:57.896 INFO:teuthology.orchestra.run.smithi026.stdout:47244640257 2024-09-17T07:46:57.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.3 2024-09-17T07:46:57.977 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:57.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-17T07:46:58.000 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-17T07:46:58.013 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-17T07:46:58.023 INFO:teuthology.orchestra.run.smithi026.stdout:47244640257 2024-09-17T07:46:58.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.0 2024-09-17T07:46:58.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T07:46:58.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T07:46:58.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T07:46:58.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T07:46:59.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T07:46:59.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T07:46:59.231 INFO:teuthology.orchestra.run.smithi026.stdout:47244640261 2024-09-17T07:46:59.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.1 2024-09-17T07:46:59.249 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:59.292 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:59.298 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-09-17T07:46:59.306 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640259 for osd.3 2024-09-17T07:46:59.307 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:59.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.4 2024-09-17T07:46:59.316 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:59.381 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-09-17T07:46:59.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.2 2024-09-17T07:46:59.393 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:59.408 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-17T07:46:59.419 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-09-17T07:46:59.421 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640259 for osd.0 2024-09-17T07:46:59.421 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:59.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.5 2024-09-17T07:46:59.432 DEBUG:teuthology.parallel:result is None 2024-09-17T07:46:59.432 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T07:46:59.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:46:59.740 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:46:59.752 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-17T07:46:59.410433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1811668,"kb_used_data":2032,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564419372,"statfs":{"total":579820584960,"available":577965436928,"internally_reserved":0,"allocated":2080768,"data_stored":1444666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000352"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.313463+0000","last_change":"2024-09-17T07:46:55.313728+0000","last_active":"2024-09-17T07:46:55.313463+0000","last_peered":"2024-09-17T07:46:55.313463+0000","last_clean":"2024-09-17T07:46:55.313463+0000","last_became_active":"2024-09-17T07:46:53.114488+0000","last_became_peered":"2024-09-17T07:46:53.114488+0000","last_unstale":"2024-09-17T07:46:55.313463+0000","last_undegraded":"2024-09-17T07:46:55.313463+0000","last_fullsized":"2024-09-17T07:46:55.313463+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:03.738226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.000334+0000","last_change":"2024-09-17T07:46:55.000533+0000","last_active":"2024-09-17T07:46:55.000334+0000","last_peered":"2024-09-17T07:46:55.000334+0000","last_clean":"2024-09-17T07:46:55.000334+0000","last_became_active":"2024-09-17T07:46:52.996044+0000","last_became_peered":"2024-09-17T07:46:52.996044+0000","last_unstale":"2024-09-17T07:46:55.000334+0000","last_undegraded":"2024-09-17T07:46:55.000334+0000","last_fullsized":"2024-09-17T07:46:55.000334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.337315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031775500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.002956+0000","last_change":"2024-09-17T07:46:55.003172+0000","last_active":"2024-09-17T07:46:55.002956+0000","last_peered":"2024-09-17T07:46:55.002956+0000","last_clean":"2024-09-17T07:46:55.002956+0000","last_became_active":"2024-09-17T07:46:53.114521+0000","last_became_peered":"2024-09-17T07:46:53.114521+0000","last_unstale":"2024-09-17T07:46:55.002956+0000","last_undegraded":"2024-09-17T07:46:55.002956+0000","last_fullsized":"2024-09-17T07:46:55.002956+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:46.450177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037727799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:56.024422+0000","last_change":"2024-09-17T07:46:56.024669+0000","last_active":"2024-09-17T07:46:56.024422+0000","last_peered":"2024-09-17T07:46:56.024422+0000","last_clean":"2024-09-17T07:46:56.024422+0000","last_became_active":"2024-09-17T07:46:52.995203+0000","last_became_peered":"2024-09-17T07:46:52.995203+0000","last_unstale":"2024-09-17T07:46:56.024422+0000","last_undegraded":"2024-09-17T07:46:56.024422+0000","last_fullsized":"2024-09-17T07:46:56.024422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:50:58.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"13'72","reported_seq":100,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-17T07:46:53.994230+0000","last_change":"2024-09-17T07:46:52.996274+0000","last_active":"2024-09-17T07:46:53.994230+0000","last_peered":"2024-09-17T07:46:53.994230+0000","last_clean":"2024-09-17T07:46:53.994230+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:46:53.994230+0000","last_undegraded":"2024-09-17T07:46:53.994230+0000","last_fullsized":"2024-09-17T07:46:53.994230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:03.825313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:56.024318+0000","last_change":"2024-09-17T07:46:56.024711+0000","last_active":"2024-09-17T07:46:56.024318+0000","last_peered":"2024-09-17T07:46:56.024318+0000","last_clean":"2024-09-17T07:46:56.024318+0000","last_became_active":"2024-09-17T07:46:52.998238+0000","last_became_peered":"2024-09-17T07:46:52.998238+0000","last_unstale":"2024-09-17T07:46:56.024318+0000","last_undegraded":"2024-09-17T07:46:56.024318+0000","last_fullsized":"2024-09-17T07:46:56.024318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:52.906241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.003565+0000","last_change":"2024-09-17T07:46:55.003565+0000","last_active":"2024-09-17T07:46:55.003565+0000","last_peered":"2024-09-17T07:46:55.003565+0000","last_clean":"2024-09-17T07:46:55.003565+0000","last_became_active":"2024-09-17T07:46:52.999808+0000","last_became_peered":"2024-09-17T07:46:52.999808+0000","last_unstale":"2024-09-17T07:46:55.003565+0000","last_undegraded":"2024-09-17T07:46:55.003565+0000","last_fullsized":"2024-09-17T07:46:55.003565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:09.551569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.313494+0000","last_change":"2024-09-17T07:46:55.313717+0000","last_active":"2024-09-17T07:46:55.313494+0000","last_peered":"2024-09-17T07:46:55.313494+0000","last_clean":"2024-09-17T07:46:55.313494+0000","last_became_active":"2024-09-17T07:46:52.998992+0000","last_became_peered":"2024-09-17T07:46:52.998992+0000","last_unstale":"2024-09-17T07:46:55.313494+0000","last_undegraded":"2024-09-17T07:46:55.313494+0000","last_fullsized":"2024-09-17T07:46:55.313494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:06:45.637243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.000296+0000","last_change":"2024-09-17T07:46:55.000486+0000","last_active":"2024-09-17T07:46:55.000296+0000","last_peered":"2024-09-17T07:46:55.000296+0000","last_clean":"2024-09-17T07:46:55.000296+0000","last_became_active":"2024-09-17T07:46:52.996744+0000","last_became_peered":"2024-09-17T07:46:52.996744+0000","last_unstale":"2024-09-17T07:46:55.000296+0000","last_undegraded":"2024-09-17T07:46:55.000296+0000","last_fullsized":"2024-09-17T07:46:55.000296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:55:26.516101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":147456,"data_stored":43982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301676,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070164,"statfs":{"total":96636764160,"available":96327847936,"internally_reserved":0,"allocated":151552,"data_stored":44001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302492,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069348,"statfs":{"total":96636764160,"available":96327012352,"internally_reserved":0,"allocated":741376,"data_stored":634350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":147456,"data_stored":43982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302492,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069348,"statfs":{"total":96636764160,"available":96327012352,"internally_reserved":0,"allocated":741376,"data_stored":634350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301672,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070168,"statfs":{"total":96636764160,"available":96327852032,"internally_reserved":0,"allocated":151552,"data_stored":44001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.085}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:46:59.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:47:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:47:00.040 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-17T07:46:59.410433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1811668,"kb_used_data":2032,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564419372,"statfs":{"total":579820584960,"available":577965436928,"internally_reserved":0,"allocated":2080768,"data_stored":1444666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000352"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.313463+0000","last_change":"2024-09-17T07:46:55.313728+0000","last_active":"2024-09-17T07:46:55.313463+0000","last_peered":"2024-09-17T07:46:55.313463+0000","last_clean":"2024-09-17T07:46:55.313463+0000","last_became_active":"2024-09-17T07:46:53.114488+0000","last_became_peered":"2024-09-17T07:46:53.114488+0000","last_unstale":"2024-09-17T07:46:55.313463+0000","last_undegraded":"2024-09-17T07:46:55.313463+0000","last_fullsized":"2024-09-17T07:46:55.313463+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:03.738226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.000334+0000","last_change":"2024-09-17T07:46:55.000533+0000","last_active":"2024-09-17T07:46:55.000334+0000","last_peered":"2024-09-17T07:46:55.000334+0000","last_clean":"2024-09-17T07:46:55.000334+0000","last_became_active":"2024-09-17T07:46:52.996044+0000","last_became_peered":"2024-09-17T07:46:52.996044+0000","last_unstale":"2024-09-17T07:46:55.000334+0000","last_undegraded":"2024-09-17T07:46:55.000334+0000","last_fullsized":"2024-09-17T07:46:55.000334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.337315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031775500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.002956+0000","last_change":"2024-09-17T07:46:55.003172+0000","last_active":"2024-09-17T07:46:55.002956+0000","last_peered":"2024-09-17T07:46:55.002956+0000","last_clean":"2024-09-17T07:46:55.002956+0000","last_became_active":"2024-09-17T07:46:53.114521+0000","last_became_peered":"2024-09-17T07:46:53.114521+0000","last_unstale":"2024-09-17T07:46:55.002956+0000","last_undegraded":"2024-09-17T07:46:55.002956+0000","last_fullsized":"2024-09-17T07:46:55.002956+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:46.450177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037727799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:56.024422+0000","last_change":"2024-09-17T07:46:56.024669+0000","last_active":"2024-09-17T07:46:56.024422+0000","last_peered":"2024-09-17T07:46:56.024422+0000","last_clean":"2024-09-17T07:46:56.024422+0000","last_became_active":"2024-09-17T07:46:52.995203+0000","last_became_peered":"2024-09-17T07:46:52.995203+0000","last_unstale":"2024-09-17T07:46:56.024422+0000","last_undegraded":"2024-09-17T07:46:56.024422+0000","last_fullsized":"2024-09-17T07:46:56.024422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:50:58.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"13'72","reported_seq":100,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-17T07:46:53.994230+0000","last_change":"2024-09-17T07:46:52.996274+0000","last_active":"2024-09-17T07:46:53.994230+0000","last_peered":"2024-09-17T07:46:53.994230+0000","last_clean":"2024-09-17T07:46:53.994230+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:46:53.994230+0000","last_undegraded":"2024-09-17T07:46:53.994230+0000","last_fullsized":"2024-09-17T07:46:53.994230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:03.825313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:56.024318+0000","last_change":"2024-09-17T07:46:56.024711+0000","last_active":"2024-09-17T07:46:56.024318+0000","last_peered":"2024-09-17T07:46:56.024318+0000","last_clean":"2024-09-17T07:46:56.024318+0000","last_became_active":"2024-09-17T07:46:52.998238+0000","last_became_peered":"2024-09-17T07:46:52.998238+0000","last_unstale":"2024-09-17T07:46:56.024318+0000","last_undegraded":"2024-09-17T07:46:56.024318+0000","last_fullsized":"2024-09-17T07:46:56.024318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:52.906241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.003565+0000","last_change":"2024-09-17T07:46:55.003565+0000","last_active":"2024-09-17T07:46:55.003565+0000","last_peered":"2024-09-17T07:46:55.003565+0000","last_clean":"2024-09-17T07:46:55.003565+0000","last_became_active":"2024-09-17T07:46:52.999808+0000","last_became_peered":"2024-09-17T07:46:52.999808+0000","last_unstale":"2024-09-17T07:46:55.003565+0000","last_undegraded":"2024-09-17T07:46:55.003565+0000","last_fullsized":"2024-09-17T07:46:55.003565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:09.551569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.313494+0000","last_change":"2024-09-17T07:46:55.313717+0000","last_active":"2024-09-17T07:46:55.313494+0000","last_peered":"2024-09-17T07:46:55.313494+0000","last_clean":"2024-09-17T07:46:55.313494+0000","last_became_active":"2024-09-17T07:46:52.998992+0000","last_became_peered":"2024-09-17T07:46:52.998992+0000","last_unstale":"2024-09-17T07:46:55.313494+0000","last_undegraded":"2024-09-17T07:46:55.313494+0000","last_fullsized":"2024-09-17T07:46:55.313494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:06:45.637243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T07:46:55.000296+0000","last_change":"2024-09-17T07:46:55.000486+0000","last_active":"2024-09-17T07:46:55.000296+0000","last_peered":"2024-09-17T07:46:55.000296+0000","last_clean":"2024-09-17T07:46:55.000296+0000","last_became_active":"2024-09-17T07:46:52.996744+0000","last_became_peered":"2024-09-17T07:46:52.996744+0000","last_unstale":"2024-09-17T07:46:55.000296+0000","last_undegraded":"2024-09-17T07:46:55.000296+0000","last_fullsized":"2024-09-17T07:46:55.000296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:55:26.516101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":147456,"data_stored":43982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301676,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070164,"statfs":{"total":96636764160,"available":96327847936,"internally_reserved":0,"allocated":151552,"data_stored":44001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302492,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069348,"statfs":{"total":96636764160,"available":96327012352,"internally_reserved":0,"allocated":741376,"data_stored":634350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":147456,"data_stored":43982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302492,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069348,"statfs":{"total":96636764160,"available":96327012352,"internally_reserved":0,"allocated":741376,"data_stored":634350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301672,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070168,"statfs":{"total":96636764160,"available":96327852032,"internally_reserved":0,"allocated":151552,"data_stored":44001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.085}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:47:00.041 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T07:47:00.041 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T07:47:00.042 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T07:47:00.042 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T07:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T07:47:00.362 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T07:47:00.362 INFO:teuthology.run_tasks:Running task exec... 2024-09-17T07:47:00.375 INFO:teuthology.task.exec:Executing custom commands... 2024-09-17T07:47:00.375 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi098.front.sepia.ceph.com 2024-09-17T07:47:00.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-09-17T07:47:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cache' created 2024-09-17T07:47:01.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2024-09-17T07:47:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2024-09-17T07:47:02.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-09-17T07:47:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:set cache-mode for pool 'cache' to writeback 2024-09-17T07:47:03.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2024-09-17T07:47:04.493 INFO:teuthology.orchestra.run.smithi098.stderr:overlay for 'rbd' is now (or already was) 'cache' 2024-09-17T07:47:04.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-09-17T07:47:05.497 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 3 hit_set_type to bloom 2024-09-17T07:47:05.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-09-17T07:47:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 3 hit_set_count to 8 2024-09-17T07:47:06.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2024-09-17T07:47:07.505 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 3 hit_set_period to 60 2024-09-17T07:47:07.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2024-09-17T07:47:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 3 target_max_objects to 250 2024-09-17T07:47:08.523 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-17T07:47:08.549 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-17T07:47:08.549 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-17T07:47:08.550 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-17T07:47:08.550 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-17T07:47:08.855 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-17T07:47:08.855 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T07:47:08.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:47:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:09.125 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:47:09.137 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-17T07:47:08.505165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1811608,"kb_used_data":2476,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564419432,"statfs":{"total":579820584960,"available":577965498368,"internally_reserved":0,"allocated":2535424,"data_stored":1641425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014232"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.615201+0000","last_change":"2024-09-17T07:46:55.313728+0000","last_active":"2024-09-17T07:47:05.615201+0000","last_peered":"2024-09-17T07:47:05.615201+0000","last_clean":"2024-09-17T07:47:05.615201+0000","last_became_active":"2024-09-17T07:46:53.114488+0000","last_became_peered":"2024-09-17T07:46:53.114488+0000","last_unstale":"2024-09-17T07:47:05.615201+0000","last_undegraded":"2024-09-17T07:47:05.615201+0000","last_fullsized":"2024-09-17T07:47:05.615201+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:03.738226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495906+0000","last_change":"2024-09-17T07:46:55.000533+0000","last_active":"2024-09-17T07:47:05.495906+0000","last_peered":"2024-09-17T07:47:05.495906+0000","last_clean":"2024-09-17T07:47:05.495906+0000","last_became_active":"2024-09-17T07:46:52.996044+0000","last_became_peered":"2024-09-17T07:46:52.996044+0000","last_unstale":"2024-09-17T07:47:05.495906+0000","last_undegraded":"2024-09-17T07:47:05.495906+0000","last_fullsized":"2024-09-17T07:47:05.495906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.337315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031775500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925077+0000","last_change":"2024-09-17T07:46:55.003172+0000","last_active":"2024-09-17T07:47:05.925077+0000","last_peered":"2024-09-17T07:47:05.925077+0000","last_clean":"2024-09-17T07:47:05.925077+0000","last_became_active":"2024-09-17T07:46:53.114521+0000","last_became_peered":"2024-09-17T07:46:53.114521+0000","last_unstale":"2024-09-17T07:47:05.925077+0000","last_undegraded":"2024-09-17T07:47:05.925077+0000","last_fullsized":"2024-09-17T07:47:05.925077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:46.450177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037727799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496125+0000","last_change":"2024-09-17T07:46:56.024669+0000","last_active":"2024-09-17T07:47:05.496125+0000","last_peered":"2024-09-17T07:47:05.496125+0000","last_clean":"2024-09-17T07:47:05.496125+0000","last_became_active":"2024-09-17T07:46:52.995203+0000","last_became_peered":"2024-09-17T07:46:52.995203+0000","last_unstale":"2024-09-17T07:47:05.496125+0000","last_undegraded":"2024-09-17T07:47:05.496125+0000","last_fullsized":"2024-09-17T07:47:05.496125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:50:58.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496283+0000","last_change":"2024-09-17T07:47:01.487813+0000","last_active":"2024-09-17T07:47:05.496283+0000","last_peered":"2024-09-17T07:47:05.496283+0000","last_clean":"2024-09-17T07:47:05.496283+0000","last_became_active":"2024-09-17T07:47:01.487515+0000","last_became_peered":"2024-09-17T07:47:01.487515+0000","last_unstale":"2024-09-17T07:47:05.496283+0000","last_undegraded":"2024-09-17T07:47:05.496283+0000","last_fullsized":"2024-09-17T07:47:05.496283+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:46:36.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":107,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495901+0000","last_change":"2024-09-17T07:46:52.996274+0000","last_active":"2024-09-17T07:47:05.495901+0000","last_peered":"2024-09-17T07:47:05.495901+0000","last_clean":"2024-09-17T07:47:05.495901+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:47:05.495901+0000","last_undegraded":"2024-09-17T07:47:05.495901+0000","last_fullsized":"2024-09-17T07:47:05.495901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:03.825313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496072+0000","last_change":"2024-09-17T07:46:56.024711+0000","last_active":"2024-09-17T07:47:05.496072+0000","last_peered":"2024-09-17T07:47:05.496072+0000","last_clean":"2024-09-17T07:47:05.496072+0000","last_became_active":"2024-09-17T07:46:52.998238+0000","last_became_peered":"2024-09-17T07:46:52.998238+0000","last_unstale":"2024-09-17T07:47:05.496072+0000","last_undegraded":"2024-09-17T07:47:05.496072+0000","last_fullsized":"2024-09-17T07:47:05.496072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:52.906241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496101+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:47:05.496101+0000","last_peered":"2024-09-17T07:47:05.496101+0000","last_clean":"2024-09-17T07:47:05.496101+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:47:05.496101+0000","last_undegraded":"2024-09-17T07:47:05.496101+0000","last_fullsized":"2024-09-17T07:47:05.496101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:05:30.296358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925068+0000","last_change":"2024-09-17T07:46:55.003565+0000","last_active":"2024-09-17T07:47:05.925068+0000","last_peered":"2024-09-17T07:47:05.925068+0000","last_clean":"2024-09-17T07:47:05.925068+0000","last_became_active":"2024-09-17T07:46:52.999808+0000","last_became_peered":"2024-09-17T07:46:52.999808+0000","last_unstale":"2024-09-17T07:47:05.925068+0000","last_undegraded":"2024-09-17T07:47:05.925068+0000","last_fullsized":"2024-09-17T07:47:05.925068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:09.551569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925096+0000","last_change":"2024-09-17T07:47:01.488782+0000","last_active":"2024-09-17T07:47:05.925096+0000","last_peered":"2024-09-17T07:47:05.925096+0000","last_clean":"2024-09-17T07:47:05.925096+0000","last_became_active":"2024-09-17T07:47:01.488528+0000","last_became_peered":"2024-09-17T07:47:01.488528+0000","last_unstale":"2024-09-17T07:47:05.925096+0000","last_undegraded":"2024-09-17T07:47:05.925096+0000","last_fullsized":"2024-09-17T07:47:05.925096+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:08.762766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.615202+0000","last_change":"2024-09-17T07:46:55.313717+0000","last_active":"2024-09-17T07:47:05.615202+0000","last_peered":"2024-09-17T07:47:05.615202+0000","last_clean":"2024-09-17T07:47:05.615202+0000","last_became_active":"2024-09-17T07:46:52.998992+0000","last_became_peered":"2024-09-17T07:46:52.998992+0000","last_unstale":"2024-09-17T07:47:05.615202+0000","last_undegraded":"2024-09-17T07:47:05.615202+0000","last_fullsized":"2024-09-17T07:47:05.615202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:06:45.637243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496181+0000","last_change":"2024-09-17T07:47:02.509843+0000","last_active":"2024-09-17T07:47:05.496181+0000","last_peered":"2024-09-17T07:47:05.496181+0000","last_clean":"2024-09-17T07:47:05.496181+0000","last_became_active":"2024-09-17T07:47:02.509516+0000","last_became_peered":"2024-09-17T07:47:02.509516+0000","last_unstale":"2024-09-17T07:47:05.496181+0000","last_undegraded":"2024-09-17T07:47:05.496181+0000","last_fullsized":"2024-09-17T07:47:05.496181+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:42.399749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495942+0000","last_change":"2024-09-17T07:46:55.000486+0000","last_active":"2024-09-17T07:47:05.495942+0000","last_peered":"2024-09-17T07:47:05.495942+0000","last_clean":"2024-09-17T07:47:05.495942+0000","last_became_active":"2024-09-17T07:46:52.996744+0000","last_became_peered":"2024-09-17T07:46:52.996744+0000","last_unstale":"2024-09-17T07:47:05.495942+0000","last_undegraded":"2024-09-17T07:47:05.495942+0000","last_fullsized":"2024-09-17T07:47:05.495942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:55:26.516101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301736,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070104,"statfs":{"total":96636764160,"available":96327786496,"internally_reserved":0,"allocated":221184,"data_stored":75894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301756,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070084,"statfs":{"total":96636764160,"available":96327766016,"internally_reserved":0,"allocated":237568,"data_stored":81200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302320,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069520,"statfs":{"total":96636764160,"available":96327188480,"internally_reserved":0,"allocated":815104,"data_stored":666262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301736,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070104,"statfs":{"total":96636764160,"available":96327786496,"internally_reserved":0,"allocated":221184,"data_stored":75894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302320,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069520,"statfs":{"total":96636764160,"available":96327188480,"internally_reserved":0,"allocated":815104,"data_stored":666262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301740,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070100,"statfs":{"total":96636764160,"available":96327782400,"internally_reserved":0,"allocated":225280,"data_stored":75913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:47:09.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:47:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:09.441 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:47:09.442 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-17T07:47:08.505165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1811608,"kb_used_data":2476,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564419432,"statfs":{"total":579820584960,"available":577965498368,"internally_reserved":0,"allocated":2535424,"data_stored":1641425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014232"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.615201+0000","last_change":"2024-09-17T07:46:55.313728+0000","last_active":"2024-09-17T07:47:05.615201+0000","last_peered":"2024-09-17T07:47:05.615201+0000","last_clean":"2024-09-17T07:47:05.615201+0000","last_became_active":"2024-09-17T07:46:53.114488+0000","last_became_peered":"2024-09-17T07:46:53.114488+0000","last_unstale":"2024-09-17T07:47:05.615201+0000","last_undegraded":"2024-09-17T07:47:05.615201+0000","last_fullsized":"2024-09-17T07:47:05.615201+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:03.738226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495906+0000","last_change":"2024-09-17T07:46:55.000533+0000","last_active":"2024-09-17T07:47:05.495906+0000","last_peered":"2024-09-17T07:47:05.495906+0000","last_clean":"2024-09-17T07:47:05.495906+0000","last_became_active":"2024-09-17T07:46:52.996044+0000","last_became_peered":"2024-09-17T07:46:52.996044+0000","last_unstale":"2024-09-17T07:47:05.495906+0000","last_undegraded":"2024-09-17T07:47:05.495906+0000","last_fullsized":"2024-09-17T07:47:05.495906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.337315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031775500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925077+0000","last_change":"2024-09-17T07:46:55.003172+0000","last_active":"2024-09-17T07:47:05.925077+0000","last_peered":"2024-09-17T07:47:05.925077+0000","last_clean":"2024-09-17T07:47:05.925077+0000","last_became_active":"2024-09-17T07:46:53.114521+0000","last_became_peered":"2024-09-17T07:46:53.114521+0000","last_unstale":"2024-09-17T07:47:05.925077+0000","last_undegraded":"2024-09-17T07:47:05.925077+0000","last_fullsized":"2024-09-17T07:47:05.925077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:46.450177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037727799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496125+0000","last_change":"2024-09-17T07:46:56.024669+0000","last_active":"2024-09-17T07:47:05.496125+0000","last_peered":"2024-09-17T07:47:05.496125+0000","last_clean":"2024-09-17T07:47:05.496125+0000","last_became_active":"2024-09-17T07:46:52.995203+0000","last_became_peered":"2024-09-17T07:46:52.995203+0000","last_unstale":"2024-09-17T07:47:05.496125+0000","last_undegraded":"2024-09-17T07:47:05.496125+0000","last_fullsized":"2024-09-17T07:47:05.496125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:50:58.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496283+0000","last_change":"2024-09-17T07:47:01.487813+0000","last_active":"2024-09-17T07:47:05.496283+0000","last_peered":"2024-09-17T07:47:05.496283+0000","last_clean":"2024-09-17T07:47:05.496283+0000","last_became_active":"2024-09-17T07:47:01.487515+0000","last_became_peered":"2024-09-17T07:47:01.487515+0000","last_unstale":"2024-09-17T07:47:05.496283+0000","last_undegraded":"2024-09-17T07:47:05.496283+0000","last_fullsized":"2024-09-17T07:47:05.496283+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:46:36.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":107,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495901+0000","last_change":"2024-09-17T07:46:52.996274+0000","last_active":"2024-09-17T07:47:05.495901+0000","last_peered":"2024-09-17T07:47:05.495901+0000","last_clean":"2024-09-17T07:47:05.495901+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:47:05.495901+0000","last_undegraded":"2024-09-17T07:47:05.495901+0000","last_fullsized":"2024-09-17T07:47:05.495901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:03.825313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496072+0000","last_change":"2024-09-17T07:46:56.024711+0000","last_active":"2024-09-17T07:47:05.496072+0000","last_peered":"2024-09-17T07:47:05.496072+0000","last_clean":"2024-09-17T07:47:05.496072+0000","last_became_active":"2024-09-17T07:46:52.998238+0000","last_became_peered":"2024-09-17T07:46:52.998238+0000","last_unstale":"2024-09-17T07:47:05.496072+0000","last_undegraded":"2024-09-17T07:47:05.496072+0000","last_fullsized":"2024-09-17T07:47:05.496072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:52.906241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496101+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:47:05.496101+0000","last_peered":"2024-09-17T07:47:05.496101+0000","last_clean":"2024-09-17T07:47:05.496101+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:47:05.496101+0000","last_undegraded":"2024-09-17T07:47:05.496101+0000","last_fullsized":"2024-09-17T07:47:05.496101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:05:30.296358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925068+0000","last_change":"2024-09-17T07:46:55.003565+0000","last_active":"2024-09-17T07:47:05.925068+0000","last_peered":"2024-09-17T07:47:05.925068+0000","last_clean":"2024-09-17T07:47:05.925068+0000","last_became_active":"2024-09-17T07:46:52.999808+0000","last_became_peered":"2024-09-17T07:46:52.999808+0000","last_unstale":"2024-09-17T07:47:05.925068+0000","last_undegraded":"2024-09-17T07:47:05.925068+0000","last_fullsized":"2024-09-17T07:47:05.925068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:09.551569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925096+0000","last_change":"2024-09-17T07:47:01.488782+0000","last_active":"2024-09-17T07:47:05.925096+0000","last_peered":"2024-09-17T07:47:05.925096+0000","last_clean":"2024-09-17T07:47:05.925096+0000","last_became_active":"2024-09-17T07:47:01.488528+0000","last_became_peered":"2024-09-17T07:47:01.488528+0000","last_unstale":"2024-09-17T07:47:05.925096+0000","last_undegraded":"2024-09-17T07:47:05.925096+0000","last_fullsized":"2024-09-17T07:47:05.925096+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:08.762766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.615202+0000","last_change":"2024-09-17T07:46:55.313717+0000","last_active":"2024-09-17T07:47:05.615202+0000","last_peered":"2024-09-17T07:47:05.615202+0000","last_clean":"2024-09-17T07:47:05.615202+0000","last_became_active":"2024-09-17T07:46:52.998992+0000","last_became_peered":"2024-09-17T07:46:52.998992+0000","last_unstale":"2024-09-17T07:47:05.615202+0000","last_undegraded":"2024-09-17T07:47:05.615202+0000","last_fullsized":"2024-09-17T07:47:05.615202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:06:45.637243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496181+0000","last_change":"2024-09-17T07:47:02.509843+0000","last_active":"2024-09-17T07:47:05.496181+0000","last_peered":"2024-09-17T07:47:05.496181+0000","last_clean":"2024-09-17T07:47:05.496181+0000","last_became_active":"2024-09-17T07:47:02.509516+0000","last_became_peered":"2024-09-17T07:47:02.509516+0000","last_unstale":"2024-09-17T07:47:05.496181+0000","last_undegraded":"2024-09-17T07:47:05.496181+0000","last_fullsized":"2024-09-17T07:47:05.496181+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:42.399749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495942+0000","last_change":"2024-09-17T07:46:55.000486+0000","last_active":"2024-09-17T07:47:05.495942+0000","last_peered":"2024-09-17T07:47:05.495942+0000","last_clean":"2024-09-17T07:47:05.495942+0000","last_became_active":"2024-09-17T07:46:52.996744+0000","last_became_peered":"2024-09-17T07:46:52.996744+0000","last_unstale":"2024-09-17T07:47:05.495942+0000","last_undegraded":"2024-09-17T07:47:05.495942+0000","last_fullsized":"2024-09-17T07:47:05.495942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:55:26.516101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301736,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070104,"statfs":{"total":96636764160,"available":96327786496,"internally_reserved":0,"allocated":221184,"data_stored":75894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301756,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070084,"statfs":{"total":96636764160,"available":96327766016,"internally_reserved":0,"allocated":237568,"data_stored":81200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302320,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069520,"statfs":{"total":96636764160,"available":96327188480,"internally_reserved":0,"allocated":815104,"data_stored":666262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301736,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070104,"statfs":{"total":96636764160,"available":96327786496,"internally_reserved":0,"allocated":221184,"data_stored":75894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302320,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069520,"statfs":{"total":96636764160,"available":96327188480,"internally_reserved":0,"allocated":815104,"data_stored":666262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301740,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070100,"statfs":{"total":96636764160,"available":96327782400,"internally_reserved":0,"allocated":225280,"data_stored":75913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:47:09.444 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T07:47:09.444 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 24 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 666af130-c9e0-433a-95ce-0b7c815e9e68 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-09-17T07:46:41.695689+0000 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-09-17T07:47:08.503039+0000 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:crush_version 4 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:full_ratio 0.95 2024-09-17T07:47:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:backfillfull_ratio 0.9 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:nearfull_ratio 0.85 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:require_min_compat_client luminous 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:min_compat_client jewel 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:require_osd_release quincy 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:stretch_mode_enabled false 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 20 lfor 20/20/20 flags hashpspool,selfmanaged_snaps tiers 3 read_tier 3 write_tier 3 stripe_width 0 application rbd 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:pool 2 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:pool 3 'cache' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 24 lfor 20/20/20 flags hashpspool,incomplete_clones,selfmanaged_snaps tier_of 1 cache_mode writeback target_objects 250 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 60s x8 decay_rate 0 search_last_n 0 stripe_width 0 application rbd 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:max_osd 6 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6800/31884,v1:172.21.15.26:6801/31884] [v2:172.21.15.26:6802/31884,v1:172.21.15.26:6803/31884] exists,up 9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 up in weight 1 up_from 11 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] [v2:172.21.15.26:6810/31885,v1:172.21.15.26:6812/31885] exists,up b26da222-1990-4e5d-a449-0ff9d561090f 2024-09-17T07:47:09.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6811/31886,v1:172.21.15.26:6813/31886] [v2:172.21.15.26:6815/31886,v1:172.21.15.26:6817/31886] exists,up e6761959-1933-4d0a-a0ff-b437b983611e 2024-09-17T07:47:09.719 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6800/31604,v1:172.21.15.94:6801/31604] [v2:172.21.15.94:6802/31604,v1:172.21.15.94:6803/31604] exists,up 5166dbfd-44ec-4d31-b486-b151dfde1443 2024-09-17T07:47:09.719 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6808/31605,v1:172.21.15.94:6809/31605] [v2:172.21.15.94:6810/31605,v1:172.21.15.94:6811/31605] exists,up f7829bfc-4efc-4bc9-8c4d-423690de31fd 2024-09-17T07:47:09.719 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6816/31606,v1:172.21.15.94:6817/31606] [v2:172.21.15.94:6818/31606,v1:172.21.15.94:6819/31606] exists,up 4e0679f8-a785-4804-93d3-79b7acabe009 2024-09-17T07:47:09.729 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6800/31884,v1:172.21.15.26:6801/31884] [v2:172.21.15.26:6802/31884,v1:172.21.15.26:6803/31884] exists,up 9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'osd.1 up in weight 1 up_from 11 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] [v2:172.21.15.26:6810/31885,v1:172.21.15.26:6812/31885] exists,up b26da222-1990-4e5d-a449-0ff9d561090f', 'osd.2 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6811/31886,v1:172.21.15.26:6813/31886] [v2:172.21.15.26:6815/31886,v1:172.21.15.26:6817/31886] exists,up e6761959-1933-4d0a-a0ff-b437b983611e', 'osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6800/31604,v1:172.21.15.94:6801/31604] [v2:172.21.15.94:6802/31604,v1:172.21.15.94:6803/31604] exists,up 5166dbfd-44ec-4d31-b486-b151dfde1443', 'osd.4 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6808/31605,v1:172.21.15.94:6809/31605] [v2:172.21.15.94:6810/31605,v1:172.21.15.94:6811/31605] exists,up f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'osd.5 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6816/31606,v1:172.21.15.94:6817/31606] [v2:172.21.15.94:6818/31606,v1:172.21.15.94:6819/31606] exists,up 4e0679f8-a785-4804-93d3-79b7acabe009'] 2024-09-17T07:47:09.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-17T07:47:09.827 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:09.827 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mon.a", 2024-09-17T07:47:09.827 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster": "ceph", 2024-09-17T07:47:09.827 INFO:teuthology.orchestra.run.smithi026.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-17T07:47:09.827 INFO:teuthology.orchestra.run.smithi026.stdout: "admin_socket_mode": "", 2024-09-17T07:47:09.827 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_client_required": "cephx, none", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_cluster_required": "cephx", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_debug": "true", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_service_required": "cephx", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_supported": "cephx", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio": "true", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio_poll_ms": "250", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio_reap_max": "16", 2024-09-17T07:47:09.828 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_async_discard": "false", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_block_size": "4096", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_aio": "false", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_inflight_ios": "false", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_enable_discard": "false", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_flock_retry": "3", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_inject_crash": "0", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_ioring": "false", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_ioring_hipri": "false", 2024-09-17T07:47:09.829 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_type": "", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_alloc_size": "1048576", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_allocator": "hybrid", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_buffered_io": "true", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_check_for_zeros": "false", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_compact_log_sync": "false", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-17T07:47:09.830 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_max_log_runway": "4194304", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_max_prefetch": "1048576", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_min_flush_size": "524288", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_min_log_runway": "1048576", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_replay_recovery": "false", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-17T07:47:09.831 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_sync_write": "false", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_allocation_from_file": "true", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_allocator": "hybrid", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_async_db_compaction": "true", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-17T07:47:09.832 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_create": "true", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_db_create": "false", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_db_path": "", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_db_size": "0", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_path": "", 2024-09-17T07:47:09.833 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_preallocate_file": "false", 2024-09-17T07:47:09.834 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_size": "107374182400", 2024-09-17T07:47:09.834 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_wal_create": "false", 2024-09-17T07:47:09.834 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_wal_path": "", 2024-09-17T07:47:09.834 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_wal_size": "100663296", 2024-09-17T07:47:09.834 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs": "true", 2024-09-17T07:47:09.834 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_autotune": "true", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-17T07:47:09.835 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_size": "0", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_type": "2q", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_clone_cow": "true", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-17T07:47:09.836 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_mode": "none", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_csum_type": "crc32c", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_freelist": "false", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-17T07:47:09.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_small_allocations": "0", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_default_buffered_read": "true", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_default_buffered_write": "false", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-17T07:47:09.838 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fail_eio": "false", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-17T07:47:09.839 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mount": "false", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_umount": "false", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-17T07:47:09.840 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_ignore_data_csum": "false", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_log_op_age": "5.000000", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_alloc_size": "0", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_blob_size": "0", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-17T07:47:09.841 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_deferred_txc": "32", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_min_alloc_size": "0", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_nid_prealloc": "1024", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_retry_disk_reads": "3", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_cf": "true", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-17T07:47:09.842 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_mem": "512", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_tracing": "false", 2024-09-17T07:47:09.843 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_zero_block_detection": "false", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-17T07:47:09.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_cluster_require_signatures": "false", 2024-09-17T07:47:09.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_cluster_require_version": "2", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_require_signatures": "false", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_require_version": "2", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_service_require_signatures": "false", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_service_require_version": "2", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_sign_messages": "true", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "chdir": "", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_acl_type": "", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_asio_thread_count": "2", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_cache_mid": "0.750000", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_cache_size": "16384", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_caps_release_delay": "5", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_check_pool_perm": "true", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-17T07:47:09.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_force_sync_read": "false", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_getattr_caps": "false", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_inject_features": "", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_inject_tick_delay": "0", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_die_on_failed_remount": "false", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_dirsize_rbytes": "true", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_force_lazyio": "false", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_fs": "", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_inject_release_failure": "false", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_max_inline_size": "4096", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-17T07:47:09.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mds_namespace": "", 2024-09-17T07:47:09.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_metadata": "", 2024-09-17T07:47:09.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mount_gid": "-1", 2024-09-17T07:47:09.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mount_timeout": "300", 2024-09-17T07:47:09.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mount_uid": "-1", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mountpoint": "/", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_notify_timeout": "10", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc": "true", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_max_dirty": "104857600", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_max_objects": "1000", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_size": "209715200", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_target_dirty": "8388608", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_permissions": "true", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_quota": "true", 2024-09-17T07:47:09.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_quota_df": "true", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_readahead_max_bytes": "0", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_readahead_max_periods": "4", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_readahead_min": "131072", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_reconnect_stale": "false", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_shutdown_timeout": "30", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_snapdir": ".snap", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_tick_interval": "1", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_trace": "", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_try_dentry_invalidate": "false", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_use_faked_inos": "false", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "client_use_random_mds": "false", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_graylog": "false", 2024-09-17T07:47:09.850 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_graylog_port": "12201", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_monitors": "default=true", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_syslog": "false", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_syslog_level": "info", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "-", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_network": "", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_network_interface": "", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zlib_isal": "false", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zlib_level": "5", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zlib_winsize": "-15", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zstd_level": "1", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-17T07:47:09.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_alien_op_num_threads": "6", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location_hook": "", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location_hook_timeout": "10", 2024-09-17T07:47:09.852 INFO:teuthology.orchestra.run.smithi026.stdout: "daemonize": "false", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_alienstore": "0/5", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_allow_any_pool_priority": "false", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_asok": "1/5", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_asok_assert_abort": "false", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_asserts_on_shutdown": "true", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_auth": "20/20", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_bdev": "1/3", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_bluefs": "1/5", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_bluestore": "1/5", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_buffer": "0/1", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_ceph_exporter": "1/5", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_cephfs_mirror": "0/5", 2024-09-17T07:47:09.853 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_cephsqlite": "0/5", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_civetweb": "1/10", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_client": "0/5", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_compressor": "1/5", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_context": "0/1", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_crush": "1/1", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_crypto": "1/5", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_deliberately_leak_memory": "false", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_disable_randomized_ping": "false", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_dpdk": "1/5", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_eventtrace": "1/5", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_filer": "0/1", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_filestore": "1/3", 2024-09-17T07:47:09.854 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_finisher": "1/1", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_fuse": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_heartbeat_testing_span": "0", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_heartbeatmap": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_javaclient": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_journal": "1/3", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_journaler": "0/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_kstore": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_leveldb": "4/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_lockdep": "0/1", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mclock": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_balancer": "1/5", 2024-09-17T07:47:09.855 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_locker": "1/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_log": "1/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_log_expire": "1/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_migrator": "1/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_memdb": "4/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mgr": "2/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mgrc": "1/5", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mon": "20/20", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_monc": "0/10", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_ms": "1/1", 2024-09-17T07:47:09.856 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_none": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_objclass": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_objectcacher": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_objecter": "0/1", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_optracker": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_osd": "1/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_paxos": "20/20", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_perfcounter": "1/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_prioritycache": "1/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rados": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd_mirror": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd_pwl": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd_replay": "0/5", 2024-09-17T07:47:09.857 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_refs": "0/0", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_reserver": "1/1", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw": "1/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_datacache": "1/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_sync": "1/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rocksdb": "4/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_cache": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_cleaner": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_device": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_journal": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_lba": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_odata": "0/5", 2024-09-17T07:47:09.858 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_omap": "0/5", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_onode": "0/5", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_tm": "0/5", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_striper": "0/1", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_test": "0/5", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_throttle": "1/1", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_timer": "0/1", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_tp": "0/5", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "device_failure_prediction_mode": "none", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_graylog": "false", 2024-09-17T07:47:09.859 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_journald": "false", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_stderr": "true", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_syslog": "false", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event_tracing": "false", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_addr": "0.0.0.0", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_http_port": "9926", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_prio_limit": "5", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_sort_metrics": "true", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_stats_period": "5", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "fake_statfs_for_testing": "0", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "fatal_signal_handlers": "true", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "filer_max_purge_ops": "10", 2024-09-17T07:47:09.860 INFO:teuthology.orchestra.run.smithi026.stdout: "filer_max_truncate_ops": "128", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_apply_finisher_threads": "1", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_blackhole": "false", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_btrfs_clone_range": "true", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_btrfs_snap": "true", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_caller_concurrency": "10", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_collect_device_partition_information": "true", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_commit_timeout": "600.000000", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_inject_read_err": "false", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_omap_check": "false", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_verify_split": "false", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_dump_file": "", 2024-09-17T07:47:09.861 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fadvise": "true", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fail_eio": "true", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fd_cache_shards": "16", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fd_cache_size": "128", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fiemap": "false", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fiemap_threshold": "4096", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-17T07:47:09.862 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_inject_stall": "0", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_journal_parallel": "false", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_journal_trailing": "false", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_journal_writeahead": "false", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_kill_at": "0", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs": "0", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-17T07:47:09.863 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size": "0", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_merge_threshold": "-10", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_odsync_write": "false", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_omap_backend": "rocksdb", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_omap_backend_path": "", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-17T07:47:09.864 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_op_thread_timeout": "60", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_op_threads": "2", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_punch_hole": "false", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-17T07:47:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_ops": "50", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_seek_data_hole": "false", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_sloppy_crc": "false", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_splice": "false", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_split_multiple": "2", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_split_rand_factor": "20", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_update_to": "1000", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-17T07:47:09.866 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_enable": "true", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_xfs_extsize": "false", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_zfs_snap": "false", 2024-09-17T07:47:09.867 INFO:teuthology.orchestra.run.smithi026.stdout: "fio_dir": "/tmp/fio", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_allow_other": "true", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_atomic_o_trunc": "true", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_big_writes": "true", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_debug": "false", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_default_permissions": "false", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_disable_pagecache": "false", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_max_write": "0", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_multithreaded": "true", 2024-09-17T07:47:09.868 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_require_active_mds": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_set_user_groups": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_splice_move": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_splice_read": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_splice_write": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_use_invalidate_cb": "true", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "gss_target_name": "ceph", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_file": "", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_inject_failure": "0", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_interval": "5", 2024-09-17T07:47:09.869 INFO:teuthology.orchestra.run.smithi026.stdout: "host": "smithi026", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-17T07:47:09.870 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "inject_early_sigterm": "false", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "jaeger_tracing_enable": "false", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_aio": "true", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_align_min_size": "65536", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_block_align": "true", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_block_size": "4096", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_dio": "true", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_discard": "false", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_force_aio": "false", 2024-09-17T07:47:09.871 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_ignore_corruption": "false", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_max_write_bytes": "10485760", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_max_write_entries": "100", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_replay_from": "0", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_write_header_frequency": "0", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_zero_on_create": "false", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journaler_prefetch_periods": "10", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journaler_prezero_periods": "5", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "journaler_write_head_interval": "15", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "", 2024-09-17T07:47:09.872 INFO:teuthology.orchestra.run.smithi026.stdout: "keyfile": "", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_backend": "rocksdb", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_default_stripe_size": "65536", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_fsck_on_mount": "false", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_max_bytes": "67108864", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_max_ops": "512", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_nid_prealloc": "1024", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_onode_map_size": "1024", 2024-09-17T07:47:09.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_sync_submit_transaction": "false", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_sync_transaction": "false", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_block_size": "65536", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_bloom_size": "0", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_cache_size": "536870912", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_compact_on_mount": "false", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_compression": "false", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_log": "", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_max_open_files": "0", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_paranoid": "false", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-17T07:47:09.874 INFO:teuthology.orchestra.run.smithi026.stdout: "librados_thread_count": "2", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "lockdep": "false", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "lockdep_force_backtrace": "false", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_coarse_timestamps": "true", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_flush_on_exit": "false", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_graylog_host": "127.0.0.1", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_graylog_port": "12201", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_max_new": "1000", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_max_recent": "10000", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_stderr_prefix": "", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_stop_at_utilization": "0.970000", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": "true", 2024-09-17T07:47:09.875 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_graylog": "false", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_journald": "false", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_stderr": "false", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_syslog": "false", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "max_rotating_auth_attempts": "10", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_action_on_write_error": "1", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_alternate_name_max": "8192", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_asio_thread_count": "2", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_export_pin": "true", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_dirs": "true", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_interval": "5", 2024-09-17T07:47:09.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_interval": "10", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_max": "-1", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_max_until": "-1", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_merge_size": "50", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_midchunk": "0.300000", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_min_start": "0.200000", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_minchunk": "0.001000", 2024-09-17T07:47:09.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_mode": "0", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_need_max": "1.200000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_need_min": "0.800000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_bits": "3", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_size": "10000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_target_decay": "10.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_beacon_grace": "15.000000", 2024-09-17T07:47:09.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_beacon_interval": "4.000000", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_mid": "0.700000", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_release_free_interval": "10", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_reservation": "0.050000", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_trim_interval": "1", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_trim_threshold": "262144", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-17T07:47:09.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_client_prealloc_inos": "1000", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_connect_bootstrapping": "false", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_damage_table_max_entries": "10000", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_auth_pins": "false", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_frag": "false", 2024-09-17T07:47:09.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_scatterstat": "false", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_subtrees": "false", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_decay_halflife": "5.000000", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_default_dir_hash": "2", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_defer_session_stale": "true", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_deny_all_reconnect": "false", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_keys_per_op": "16384", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_max_commit_size": "10", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_max_entries": "0", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_on_map": "false", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-17T07:47:09.881 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_early_reply": "true", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_enable_op_tracker": "true", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_enforce_unique_name": "true", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_random": "true", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_health_summarize_threshold": "10", 2024-09-17T07:47:09.882 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_health_dummy": "false", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_migrator_session_race": "false", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_join_fs": "", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_journal_format": "1", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_create_at": "0", 2024-09-17T07:47:09.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_export_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_import_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_journal_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_journal_expire_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_journal_replay_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_link_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_mdstable_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_openc_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_rename_at": "0", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_events_per_segment": "1024", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_max_events": "-1", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_max_segments": "128", 2024-09-17T07:47:09.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_pause": "false", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_segment_size": "0", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_warn_factor": "2.000000", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_caps_per_client": "1048576", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_completed_flushes": "100000", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_completed_requests": "100000", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_export_size": "20971520", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_file_recover": "32", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_purge_files": "64", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_purge_ops": "8192", 2024-09-17T07:47:09.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_snaps_per_dir": "100", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_metrics_update_interval": "2", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_min_caps_per_client": "100", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_min_caps_working_set": "10000", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_numa_node": "-1", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_complaint_time": "30.000000", 2024-09-17T07:47:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_duration": "600", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_size": "20", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_slow_op_size": "20", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_log_threshold": "5", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_ping_grace": "15", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_ping_interval": "5", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_max_caps": "30000", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-17T07:47:09.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_warning_threshold": "262144", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_replay_interval": "1.000000", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_root_ino_gid": "0", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_root_ino_uid": "0", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-17T07:47:09.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_shutdown_check": "0", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_skip_ino": "0", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_snap_max_uid": "4294967294", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_snap_min_uid": "0", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_snap_rstat": "false", 2024-09-17T07:47:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_standby_replay_damaged": "false", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_symlink_recovery": "true", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_thrash_exports": "0", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_thrash_fragments": "0", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_tick_interval": "5.000000", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_valgrind_exit": "false", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_verify_backtrace": "1", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_verify_scatter": "false", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-17T07:47:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_wipe_sessions": "false", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mempool_debug": "false", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_device_bytes": "1073741824", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_page_set": "false", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_page_size": "65536", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_client_bytes": "134217728", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_client_messages": "512", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-17T07:47:09.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_max_pg_num_change": "128", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mds_bytes": "134217728", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mds_messages": "128", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mon_bytes": "134217728", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mon_messages": "128", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_osd_bytes": "536870912", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_osd_messages": "8192", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_pool": "true", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_standby_modules": "true", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_stats_period": "5", 2024-09-17T07:47:09.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_stats_threshold": "5", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_tick_period": "2", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_allow_pool_delete": "true", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_allow_pool_size_one": "true", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_auth_validate_all_caps": "true", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_bytes": "104857600", 2024-09-17T07:47:09.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_directed_command_retry": "5", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_parallel": "3", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_log_interval": "1.000000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_ping_interval": "10.000000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-17T07:47:09.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_level": "debug", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_file": "true", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_journald": "false", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_compact_on_bootstrap": "false", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_compact_on_start": "false", 2024-09-17T07:47:09.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_compact_on_trim": "true", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cpu_threads": "4", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_crush_min_required_version": "hammer", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_daemon_bytes": "419430400", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data_avail_crit": "5", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data_avail_warn": "5", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data_size_warn": "16106127360", 2024-09-17T07:47:09.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_dump_json": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_dump_transactions": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_extra_checks": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_require_pacific": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_require_quincy": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-17T07:47:09.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_down_mkfs_grace": "120", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_election_default_strategy": "1", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_election_timeout": "5.000000", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_elector_ping_divisor": "2", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_enable_op_tracker": "true", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_fake_pool_delete": "false", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_force_quorum_join": "false", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_globalid_prealloc": "10000", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_detail_to_clog": "false", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_log_update_period": "5", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_max_detail": "50", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_to_clog": "true", 2024-09-17T07:47:09.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_to_clog_interval": "600", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_host": "172.21.15.26,[v2:172.21.15.26:3301,v1:172.21.15.26:6790],172.21.15.94", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_host_override": "", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_initial_members": "", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_lease": "5.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-17T07:47:09.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_log_full_interval": "50", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_log_max": "10000", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_log_max_summary": "50", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_log_epochs": "500", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_osd": "10000", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_pg_per_osd": "10000", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_pool_pg_num": "65536", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-17T07:47:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mds_force_trim_to": "0", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mds_skip_sanity": "false", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_memory_autotune": "true", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_memory_target": "2147483648", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_beacon_grace": "30", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_digest_period": "5", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_inactive_grace": "60", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-17T07:47:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_min_osdmap_epochs": "500", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_complaint_time": "30", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_duration": "600", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_size": "20", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_slow_op_size": "20", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_log_threshold": "5", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_auto_mark_in": "false", 2024-09-17T07:47:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_cache_size": "500", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_down_out_interval": "600", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_force_trim_to": "0", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-17T07:47:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_min_down_reporters": "2", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-17T07:47:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_report_timeout": "900", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-17T07:47:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_stuck_threshold": "60", 2024-09-17T07:47:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-17T07:47:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-17T07:47:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-17T07:47:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-17T07:47:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-17T07:47:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-17T07:47:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_probe_timeout": "2.000000", 2024-09-17T07:47:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_max_change": "0.050000", 2024-09-17T07:47:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_max_osds": "4", 2024-09-17T07:47:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-17T07:47:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-17T07:47:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-17T07:47:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-17T07:47:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-17T07:47:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_interval": "86400", 2024-09-17T07:47:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_max_keys": "100", 2024-09-17T07:47:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_timeout": "300", 2024-09-17T07:47:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_session_timeout": "300", 2024-09-17T07:47:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_smart_report_timeout": "5", 2024-09-17T07:47:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stat_smooth_intervals": "6", 2024-09-17T07:47:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-17T07:47:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_pool_min_size": "2", 2024-09-17T07:47:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_pool_size": "4", 2024-09-17T07:47:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-17T07:47:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-17T07:47:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_debug": "false", 2024-09-17T07:47:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-17T07:47:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-17T07:47:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_provider_kill_at": "0", 2024-09-17T07:47:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_requester_kill_at": "0", 2024-09-17T07:47:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_timeout": "60.000000", 2024-09-17T07:47:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_target_pg_per_osd": "100", 2024-09-17T07:47:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_tick_interval": "5", 2024-09-17T07:47:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_timecheck_interval": "300.000000", 2024-09-17T07:47:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-17T07:47:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_older_version_delay": "604800", 2024-09-17T07:47:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-17T07:47:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-17T07:47:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-17T07:47:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-17T07:47:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-17T07:47:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-17T07:47:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-17T07:47:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_misplaced": "false", 2024-09-17T07:47:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-17T07:47:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_older_version": "true", 2024-09-17T07:47:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-17T07:47:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-17T07:47:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-17T07:47:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-17T07:47:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-17T07:47:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-17T07:47:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-17T07:47:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-17T07:47:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-17T07:47:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": "", 2024-09-17T07:47:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_op_threads": "3", 2024-09-17T07:47:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-17T07:47:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_cm": "false", 2024-09-17T07:47:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_device_name": "", 2024-09-17T07:47:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_dscp": "96", 2024-09-17T07:47:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-17T07:47:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-17T07:47:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_local_gid": "", 2024-09-17T07:47:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-17T07:47:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_port_num": "1", 2024-09-17T07:47:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-17T07:47:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-17T07:47:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-17T07:47:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-17T07:47:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_sl": "3", 2024-09-17T07:47:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_support_srq": "true", 2024-09-17T07:47:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_type": "ib", 2024-09-17T07:47:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_reap_threshold": "5", 2024-09-17T07:47:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_before_connect": "false", 2024-09-17T07:47:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_ipv4": "true", 2024-09-17T07:47:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_ipv6": "false", 2024-09-17T07:47:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_msgr1": "true", 2024-09-17T07:47:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_msgr2": "true", 2024-09-17T07:47:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_port_max": "7568", 2024-09-17T07:47:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_port_min": "6800", 2024-09-17T07:47:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-17T07:47:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_retry_count": "3", 2024-09-17T07:47:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_retry_delay": "5", 2024-09-17T07:47:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_client": "false", 2024-09-17T07:47:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_mds": "false", 2024-09-17T07:47:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_mgr": "false", 2024-09-17T07:47:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_mon": "false", 2024-09-17T07:47:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_osd": "false", 2024-09-17T07:47:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_client_mode": "crc secure", 2024-09-17T07:47:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_cluster_mode": "crc secure", 2024-09-17T07:47:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_cluster_type": "", 2024-09-17T07:47:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_compress_secure": "false", 2024-09-17T07:47:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_connection_idle_timeout": "900", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_connection_ready_timeout": "10", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_crc_data": "true", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_crc_header": "true", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_bad_msg": "false", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_bug": "true", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_old_message": "true", 2024-09-17T07:47:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_skipped_message": "false", 2024-09-17T07:47:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-17T07:47:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-17T07:47:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_coremask": "0xF", 2024-09-17T07:47:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-17T07:47:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-17T07:47:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_enable_tso": "true", 2024-09-17T07:47:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-17T07:47:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-17T07:47:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_hugepages": "", 2024-09-17T07:47:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-17T07:47:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-17T07:47:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_lro": "true", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_memory_channel": "4", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_pmd": "", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_port_id": "0", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dump_on_send": "false", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_initial_backoff": "0.200000", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_delay_max": "1.000000", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-17T07:47:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_delay_type": "", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_socket_failures": "5000", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_learn_addr_from_peer": "true", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_max_accept_failures": "4", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_max_backoff": "15.000000", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_mon_client_mode": "secure crc", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_mon_service_mode": "secure crc", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_osd_compress_min_size": "1024", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_osd_compress_mode": "none", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_pq_min_cost": "65536", 2024-09-17T07:47:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_public_type": "", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_service_mode": "crc secure", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_listen_backlog": "512", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_nodelay": "true", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_rcvbuf": "0", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_type": "async+posix", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "no_config_file": "false", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_completion_locks_per_session": "32", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_inflight_ops": "1024", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-17T07:47:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_tick_interval": "5.000000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_timeout": "10.000000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "objectstore_blackhole": "false", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "openssl_engine_opts": "", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_delay_time": "5.000000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_hist_halflife": "1000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_max_low_ops": "2", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_max_ops": "4", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_slop": "0.020000", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-17T07:47:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_asio_thread_count": "2", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_async_recovery_min_cost": "100", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backfill_scan_max": "512", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backfill_scan_min": "64", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backoff_on_degraded": "false", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backoff_on_peering": "false", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backoff_on_unfound": "true", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_beacon_report_interval": "300", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_duration": "30", 2024-09-17T07:47:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_max_block_size": "67108864", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_blkin_trace_all": "false", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_check_for_log_corruption": "false", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_update_on_start": "true", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_message_cap": "256", 2024-09-17T07:47:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_message_size_cap": "524288000", 2024-09-17T07:47:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_op_priority": "63", 2024-09-17T07:47:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_watch_timeout": "30", 2024-09-17T07:47:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_command_max_records": "256", 2024-09-17T07:47:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_compact_on_start": "false", 2024-09-17T07:47:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-17T07:47:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_update_on_start": "true", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_update_weight_set": "true", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-17T07:47:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_feed_pullee": "-1", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_misdirected_ops": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_no_acting_change": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_no_purge_strays": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_op_order": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-17T07:47:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_shutdown": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_snaps": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_keys": "1024", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-17T07:47:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_stride": "524288", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_default_notify_timeout": "30", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep": "0.000000", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_discard_disconnected_ops": "true", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_enable_op_tracker": "true", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-17T07:47:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_info": "true", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_shutdown": "true", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_function_tracing": "false", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_grace": "20", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_interval": "6", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_min_peers": "10", 2024-09-17T07:47:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_min_size": "2000", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_stale": "600", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_hit_set_max_size": "100000", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_hit_set_min_size": "1000", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_journal_size": "5120", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_kill_backfill_at": "0", 2024-09-17T07:47:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_cache_size": "50", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_dedup": "true", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_message_max": "40", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_share_max_epochs": "40", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_attr_name_len": "100", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_attr_size": "0", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_backfills": "1", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_markdown_count": "5", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_markdown_period": "600", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_object_name_len": "2048", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_object_namespace_len": "256", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_object_size": "134217728", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-17T07:47:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pg_blocked_by": "16", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pg_log_entries": "10000", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pgls": "1024", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_push_cost": "8388608", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_push_objects": "10", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_scrubs": "3", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_trimming_pgs": "2", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_write_op_reply_len": "64", 2024-09-17T07:47:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_write_size": "90", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_profile": "balanced", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-17T07:47:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_base": "805306368", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_cache_min": "134217728", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-17T07:47:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_target": "4294967296", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_target_autotune": "false", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_min_pg_log_entries": "250", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_min_recovery_priority": "0", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_report_interval": "5", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_num_cache_shards": "32", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_num_op_tracker_shard": "32", 2024-09-17T07:47:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_numa_auto_affinity": "true", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_numa_node": "-1", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_numa_prefer_iface": "true", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objecter_finishers": "1", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objectstore": "bluestore", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objectstore_fuse": "false", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objectstore_tracing": "false", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_complaint_time": "30.000000", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_duration": "600", 2024-09-17T07:47:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_size": "20", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_slow_op_size": "20", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_log_threshold": "5", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_shards": "0", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_shards_hdd": "5", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_shards_ssd": "8", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_pq_min_cost": "65536", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-17T07:47:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_queue_cut_off": "high", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_thread_timeout": "15", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_open_classes_on_start": "true", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_os_flags": "0", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_peering_op_priority": "255", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_delete_cost": "1048576", 2024-09-17T07:47:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_delete_priority": "5", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_log_trim_max": "10000", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_log_trim_min": "100", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-17T07:47:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-17T07:47:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-17T07:47:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-17T07:47:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-17T07:47:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-17T07:47:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-17T07:47:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-17T07:47:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-17T07:47:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-17T07:47:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-17T07:47:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flags": "0", 2024-09-17T07:47:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-17T07:47:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_min_size": "0", 2024-09-17T07:47:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_pg_num": "32", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_pgp_num": "0", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_size": "2", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_type": "replicated", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-17T07:47:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_push_per_object_cost": "1000", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recover_clone_overlap": "true", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_cost": "20971520", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-17T07:47:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_active": "0", 2024-09-17T07:47:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-17T07:47:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-17T07:47:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-17T07:47:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-17T07:47:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_single_start": "1", 2024-09-17T07:47:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_op_priority": "3", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_priority": "5", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep": "0.000000", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-17T07:47:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_repair_during_recovery": "false", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_requested_scrub_priority": "120", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_auto_repair": "false", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-17T07:47:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_begin_hour": "0", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_begin_week_day": "0", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_chunk_max": "25", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_chunk_min": "5", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_cost": "52428800", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_during_recovery": "false", 2024-09-17T07:47:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_end_hour": "0", 2024-09-17T07:47:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_end_week_day": "0", 2024-09-17T07:47:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_event_cost": "4096", 2024-09-17T07:47:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-17T07:47:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-17T07:47:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_invalid_stats": "true", 2024-09-17T07:47:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_max_preemptions": "5", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_priority": "5", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_sleep": "0.000000", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-17T07:47:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_skip_data_digest": "false", 2024-09-17T07:47:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_smart_report_timeout": "5", 2024-09-17T07:47:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_cost": "1048576", 2024-09-17T07:47:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_priority": "5", 2024-09-17T07:47:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-17T07:47:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-17T07:47:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-17T07:47:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-17T07:47:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-17T07:47:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_target_transaction_size": "30", 2024-09-17T07:47:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-17T07:47:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-17T07:47:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tracing": "false", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_use_stale_snap": "false", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osdc_blkin_trace_all": "false", 2024-09-17T07:47:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_kill_at": "0", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_max_join_drift": "10", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_min": "500", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_min_wait": "0.050000", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_propose_interval": "1.000000", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_service_trim_max": "500", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_service_trim_min": "250", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_stash_full_interval": "25", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_trim_max": "500", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_trim_min": "250", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "perf": "true", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-17T07:47:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "-", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrv": "", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_bind_addr": "-", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_network": "", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_network_interface": "", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "qat_compressor_enabled": "false", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rados_mon_op_timeout": "0", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rados_osd_op_timeout": "0", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rados_tracing": "false", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_atime_update_interval": "60", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_balance_parent_reads": "false", 2024-09-17T07:47:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_balance_snap_reads": "false", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_blkin_trace_all": "false", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache": "true", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_policy": "writearound", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_size": "33554432", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-17T07:47:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_clone_copy_on_read": "false", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_compression_hint": "none", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_concurrent_management_ops": "10", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_clone_format": "auto", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_data_pool": "", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_features": "61", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_format": "2", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_map_options": "", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_order": "22", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_pool": "rbd", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-17T07:47:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_stripe_count": "0", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_stripe_unit": "0", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_enable_alloc_hint": "true", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_io_scheduler": "simple", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-17T07:47:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_order": "24", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_pool": "", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_splay_width": "4", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_localize_parent_reads": "false", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_localize_snap_reads": "false", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-17T07:47:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-17T07:47:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-17T07:47:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mtime_update_interval": "60", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_non_blocking_aio": "true", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_op_thread_timeout": "60", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_op_threads": "1", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_parent_cache_enabled": "false", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-17T07:47:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_plugins": "", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_bps_burst": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_bps_limit": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_exclude_ops": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_iops_burst": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_iops_limit": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-17T07:47:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-17T07:47:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_read_from_replica_policy": "default", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_skip_partial_discard": "true", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_tracing": "false", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_validate_names": "true", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_validate_pool": "true", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "restapi_base_url": "", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "restapi_log_level": "", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_acl_grants_max_num": "100", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_admin_entry": "admin", 2024-09-17T07:47:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_backend_store": "rados", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_barbican_url": "", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_beast_enable_async": "true", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cache_enabled": "true", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cache_expiry_interval": "900", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cache_lru_size": "10000", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_content_length_compat": "false", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_copy_obj_progress": "true", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-17T07:47:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cors_rules_max_num": "100", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cross_domain_policy": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_addr": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_password": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_username": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_require_ssl": "true", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-17T07:47:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_addr": "", 2024-09-17T07:47:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_auth": "token", 2024-09-17T07:47:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_namespace": "", 2024-09-17T07:47:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_prefix": "", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_token_file": "", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_buffersize": "524288", 2024-09-17T07:47:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_low_speed_time": "300", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_changes_size": "1000", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_num_shards": "128", 2024-09-17T07:47:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_window": "30", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-17T07:47:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dns_name": "", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dns_s3website_name": "", 2024-09-17T07:47:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dynamic_resharding": "true", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_gc_threads": "true", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_lc_threads": "true", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_ops_log": "false", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_quota_threads": "true", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_static_website": "false", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_usage_log": "false", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enforce_swift_acls": "true", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_exit_timeout_secs": "120", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_expose_bucket": "false", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_extended_http_attrs": "", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_frontends": "beast port=7480", 2024-09-17T07:47:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_deferred": "50", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_objs": "32", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_processor_period": "3600", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_init_timeout": "300", 2024-09-17T07:47:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_kafka_connection_idle": "300", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_domain": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_password": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_password_path": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_project": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_tenant": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_token": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_token_path": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_user": "", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_api_version": "2", 2024-09-17T07:47:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_domain": "", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_password": "", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_project": "", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_user": "", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_url": "", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_debug_interval": "-1", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_lock_max_time": "90", 2024-09-17T07:47:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_objs": "32", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_rules": "1000", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_worker": "3", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_thread_delay": "0", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_dnattr": "uid", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_searchfilter": "", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_uri": "", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-17T07:47:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_http_headers": "", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_object_name_utc": "false", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_attr_name_len": "0", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_attr_size": "0", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-17T07:47:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_chunk_size": "4194304", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_listing_results": "1000", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_notify_retries": "10", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_put_param_size": "1048576", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_put_size": "5368709120", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_slo_entries": "1000", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_md_log_max_shards": "64", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_mp_lock_max_time": "600", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-17T07:47:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_max_gc": "300", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-17T07:47:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_num_async_rados_threads": "32", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_num_control_oids": "8", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_numa_node": "-1", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_objexp_chunk_size": "100", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_objexp_gc_interval": "600", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_op_thread_timeout": "600", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_op_tracing": "false", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_opa_token": "", 2024-09-17T07:47:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_opa_url": "", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_opa_verify_ssl": "true", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_file_path": "", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_rados": "true", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_socket_path": "", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_push_interval": "2.000000", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_print_continue": "true", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-17T07:47:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rados_tracing": "false", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_realm": "", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_realm_id": "", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_region": "", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_request_uri": "", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_batch_size": "64", 2024-09-17T07:47:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_max_aio": "128", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_num_logs": "16", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_thread_interval": "600", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_resolve_cname": "false", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_run_sync_thread": "true", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-17T07:47:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_scheduler_type": "throttler", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_script_uri": "", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_service_provider_name": "", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_client_id": "", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_client_secret": "", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_entry": "sts", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_key": "sts", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-17T07:47:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_min_session_duration": "900", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_token_introspection_url": "", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_account_in_url": "false", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_auth_entry": "auth", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_auth_url": "", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_custom_header": "", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_need_stats": "true", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_tenant_name": "", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_token_expiration": "86400", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_url": "", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_url_prefix": "swift", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-17T07:47:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_lease_period": "120", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_thread_pool_size": "512", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_comment": "", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_createby": "", 2024-09-17T07:47:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_encoding": "", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_flag": "false", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_origin": "", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_tracker": "", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_trust_forwarded_https": "false", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_max_shards": "32", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_max_user_shards": "1", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_use_opa_authz": "false", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-17T07:47:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_max_buckets": "1000", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_unique_email": "true", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_verify_ssl": "true", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zone": "", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zone_id": "", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zonegroup": "", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zonegroup_id": "", 2024-09-17T07:47:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_block_size": "4096", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_size": "536870912", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-17T07:47:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_index_type": "binary_search", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_partition_filters": "false", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_perf": "false", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "run_dir": "/var/run/ceph", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_block_create": "true", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_cache_lru_size": "67108864", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_default_max_object_size": "16777216", 2024-09-17T07:47:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_device_size": "107374182400", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_batch_capacity": "16", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_segment_size": "67108864", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "setgroup": "", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "setuser": "", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "setuser_match_path": "", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "thp": "false", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "threadpool_default_timeout": "60", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-17T07:47:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "throttler_perf_counter": "true" 2024-09-17T07:47:10.005 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:10.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-17T07:47:10.116 INFO:teuthology.orchestra.run.smithi026.stdout:mon.a: {} 2024-09-17T07:47:10.117 INFO:teuthology.orchestra.run.smithi026.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:47:10.125 INFO:teuthology.orchestra.run.smithi026.stdout:mon.b: {} 2024-09-17T07:47:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:47:10.134 INFO:teuthology.orchestra.run.smithi026.stdout:mon.c: {} 2024-09-17T07:47:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:47:10.145 DEBUG:teuthology.orchestra.run.smithi026:> type ceph-objectstore-tool 2024-09-17T07:47:10.159 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T07:47:10.159 DEBUG:teuthology.orchestra.run.smithi094:> type ceph-objectstore-tool 2024-09-17T07:47:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T07:47:10.178 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T07:47:10.248 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:47:10.248 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T07:47:10.248 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T07:47:10.249 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-17T07:47:10.249 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:10.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:10.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:47:10.250 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-17T07:47:10.250 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-17T07:47:10.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:10.251 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-17T07:47:10.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:10.251 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-17T07:47:10.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:47:10.253 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:10.251+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:47:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-17T07:47:10.271 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T07:47:10.330 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-17T07:47:10.330 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-17T07:47:10.352 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:10.352+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:10.363 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:10.363 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:10.363 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:10.363 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:10.375 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:10.378 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:10.389 INFO:tasks.workunit:timeout=3h 2024-09-17T07:47:10.389 INFO:tasks.workunit:cleanup=True 2024-09-17T07:47:10.389 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-17T07:47:10.395 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:10.412 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:10.412 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:10.428 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:10.428 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:10.444 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:10.444 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:10.448 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T07:47:10.453 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:10.452+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:10.460 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:10.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:10.499 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:10.499 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:10.508 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:10.563 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:47:10.564 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:10.561+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:10.575 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-17T07:47:09.412180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1811608,"kb_used_data":2476,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564419432,"statfs":{"total":579820584960,"available":577965498368,"internally_reserved":0,"allocated":2535424,"data_stored":1641425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000610"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.615201+0000","last_change":"2024-09-17T07:46:55.313728+0000","last_active":"2024-09-17T07:47:05.615201+0000","last_peered":"2024-09-17T07:47:05.615201+0000","last_clean":"2024-09-17T07:47:05.615201+0000","last_became_active":"2024-09-17T07:46:53.114488+0000","last_became_peered":"2024-09-17T07:46:53.114488+0000","last_unstale":"2024-09-17T07:47:05.615201+0000","last_undegraded":"2024-09-17T07:47:05.615201+0000","last_fullsized":"2024-09-17T07:47:05.615201+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:03.738226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495906+0000","last_change":"2024-09-17T07:46:55.000533+0000","last_active":"2024-09-17T07:47:05.495906+0000","last_peered":"2024-09-17T07:47:05.495906+0000","last_clean":"2024-09-17T07:47:05.495906+0000","last_became_active":"2024-09-17T07:46:52.996044+0000","last_became_peered":"2024-09-17T07:46:52.996044+0000","last_unstale":"2024-09-17T07:47:05.495906+0000","last_undegraded":"2024-09-17T07:47:05.495906+0000","last_fullsized":"2024-09-17T07:47:05.495906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.337315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031775500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925077+0000","last_change":"2024-09-17T07:46:55.003172+0000","last_active":"2024-09-17T07:47:05.925077+0000","last_peered":"2024-09-17T07:47:05.925077+0000","last_clean":"2024-09-17T07:47:05.925077+0000","last_became_active":"2024-09-17T07:46:53.114521+0000","last_became_peered":"2024-09-17T07:46:53.114521+0000","last_unstale":"2024-09-17T07:47:05.925077+0000","last_undegraded":"2024-09-17T07:47:05.925077+0000","last_fullsized":"2024-09-17T07:47:05.925077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:46.450177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037727799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496125+0000","last_change":"2024-09-17T07:46:56.024669+0000","last_active":"2024-09-17T07:47:05.496125+0000","last_peered":"2024-09-17T07:47:05.496125+0000","last_clean":"2024-09-17T07:47:05.496125+0000","last_became_active":"2024-09-17T07:46:52.995203+0000","last_became_peered":"2024-09-17T07:46:52.995203+0000","last_unstale":"2024-09-17T07:47:05.496125+0000","last_undegraded":"2024-09-17T07:47:05.496125+0000","last_fullsized":"2024-09-17T07:47:05.496125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:50:58.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496283+0000","last_change":"2024-09-17T07:47:01.487813+0000","last_active":"2024-09-17T07:47:05.496283+0000","last_peered":"2024-09-17T07:47:05.496283+0000","last_clean":"2024-09-17T07:47:05.496283+0000","last_became_active":"2024-09-17T07:47:01.487515+0000","last_became_peered":"2024-09-17T07:47:01.487515+0000","last_unstale":"2024-09-17T07:47:05.496283+0000","last_undegraded":"2024-09-17T07:47:05.496283+0000","last_fullsized":"2024-09-17T07:47:05.496283+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:46:36.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":107,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495901+0000","last_change":"2024-09-17T07:46:52.996274+0000","last_active":"2024-09-17T07:47:05.495901+0000","last_peered":"2024-09-17T07:47:05.495901+0000","last_clean":"2024-09-17T07:47:05.495901+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:47:05.495901+0000","last_undegraded":"2024-09-17T07:47:05.495901+0000","last_fullsized":"2024-09-17T07:47:05.495901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:03.825313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496072+0000","last_change":"2024-09-17T07:46:56.024711+0000","last_active":"2024-09-17T07:47:05.496072+0000","last_peered":"2024-09-17T07:47:05.496072+0000","last_clean":"2024-09-17T07:47:05.496072+0000","last_became_active":"2024-09-17T07:46:52.998238+0000","last_became_peered":"2024-09-17T07:46:52.998238+0000","last_unstale":"2024-09-17T07:47:05.496072+0000","last_undegraded":"2024-09-17T07:47:05.496072+0000","last_fullsized":"2024-09-17T07:47:05.496072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:52.906241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496101+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:47:05.496101+0000","last_peered":"2024-09-17T07:47:05.496101+0000","last_clean":"2024-09-17T07:47:05.496101+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:47:05.496101+0000","last_undegraded":"2024-09-17T07:47:05.496101+0000","last_fullsized":"2024-09-17T07:47:05.496101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:05:30.296358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925068+0000","last_change":"2024-09-17T07:46:55.003565+0000","last_active":"2024-09-17T07:47:05.925068+0000","last_peered":"2024-09-17T07:47:05.925068+0000","last_clean":"2024-09-17T07:47:05.925068+0000","last_became_active":"2024-09-17T07:46:52.999808+0000","last_became_peered":"2024-09-17T07:46:52.999808+0000","last_unstale":"2024-09-17T07:47:05.925068+0000","last_undegraded":"2024-09-17T07:47:05.925068+0000","last_fullsized":"2024-09-17T07:47:05.925068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:09.551569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.925096+0000","last_change":"2024-09-17T07:47:01.488782+0000","last_active":"2024-09-17T07:47:05.925096+0000","last_peered":"2024-09-17T07:47:05.925096+0000","last_clean":"2024-09-17T07:47:05.925096+0000","last_became_active":"2024-09-17T07:47:01.488528+0000","last_became_peered":"2024-09-17T07:47:01.488528+0000","last_unstale":"2024-09-17T07:47:05.925096+0000","last_undegraded":"2024-09-17T07:47:05.925096+0000","last_fullsized":"2024-09-17T07:47:05.925096+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:08.762766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.615202+0000","last_change":"2024-09-17T07:46:55.313717+0000","last_active":"2024-09-17T07:47:05.615202+0000","last_peered":"2024-09-17T07:47:05.615202+0000","last_clean":"2024-09-17T07:47:05.615202+0000","last_became_active":"2024-09-17T07:46:52.998992+0000","last_became_peered":"2024-09-17T07:46:52.998992+0000","last_unstale":"2024-09-17T07:47:05.615202+0000","last_undegraded":"2024-09-17T07:47:05.615202+0000","last_fullsized":"2024-09-17T07:47:05.615202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:06:45.637243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.496181+0000","last_change":"2024-09-17T07:47:02.509843+0000","last_active":"2024-09-17T07:47:05.496181+0000","last_peered":"2024-09-17T07:47:05.496181+0000","last_clean":"2024-09-17T07:47:05.496181+0000","last_became_active":"2024-09-17T07:47:02.509516+0000","last_became_peered":"2024-09-17T07:47:02.509516+0000","last_unstale":"2024-09-17T07:47:05.496181+0000","last_undegraded":"2024-09-17T07:47:05.496181+0000","last_fullsized":"2024-09-17T07:47:05.496181+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:42.399749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-17T07:47:05.495942+0000","last_change":"2024-09-17T07:46:55.000486+0000","last_active":"2024-09-17T07:47:05.495942+0000","last_peered":"2024-09-17T07:47:05.495942+0000","last_clean":"2024-09-17T07:47:05.495942+0000","last_became_active":"2024-09-17T07:46:52.996744+0000","last_became_peered":"2024-09-17T07:46:52.996744+0000","last_unstale":"2024-09-17T07:47:05.495942+0000","last_undegraded":"2024-09-17T07:47:05.495942+0000","last_fullsized":"2024-09-17T07:47:05.495942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:55:26.516101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301736,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070104,"statfs":{"total":96636764160,"available":96327786496,"internally_reserved":0,"allocated":221184,"data_stored":75894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301756,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070084,"statfs":{"total":96636764160,"available":96327766016,"internally_reserved":0,"allocated":237568,"data_stored":81200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302320,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069520,"statfs":{"total":96636764160,"available":96327188480,"internally_reserved":0,"allocated":815104,"data_stored":666262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301736,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070104,"statfs":{"total":96636764160,"available":96327786496,"internally_reserved":0,"allocated":221184,"data_stored":75894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302320,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069520,"statfs":{"total":96636764160,"available":96327188480,"internally_reserved":0,"allocated":815104,"data_stored":666262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301740,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070100,"statfs":{"total":96636764160,"available":96327782400,"internally_reserved":0,"allocated":225280,"data_stored":75913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:47:10.650 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:10.650 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:10.655 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:10.654+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:10.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:10.756 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:10.755+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:10.760 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:10.760 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:10.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:10.856 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:10.855+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:10.870 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:10.870 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:10.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:10.957 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:10.956+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:11.021 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.021 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:11.058 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:11.058+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:11.130 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.130 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:11.182 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:11.181+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:11.240 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.240 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:11.260 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:11.260+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:11.361 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:11.360+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:11.390 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.390 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:11.462 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:11.462+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:11.508 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.508 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.515 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:47:11.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:11.563 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:11.561+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:11.656 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.656 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.663 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:11.663+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:11.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:11.762 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.762 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.764 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:11.764+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:11.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:11.865 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:11.864+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:11.909 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:11.909 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:11.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:11.966 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:11.965+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:12.014 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:12.014 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:12.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:12.067 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:12.066+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:12.119 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:12.119 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:12.127 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:12.168 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:12.167+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:12.224 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:12.224 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:12.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:12.269 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:12.268+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:12.331 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:12.331 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:12.339 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:12.382 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:12.382+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:12.437 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:12.438 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:12.444 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:12.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:12.485 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:12.485+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:12.559 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:12.560 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:12.570 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:12.571 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:12.571+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:12.616 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:12.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:12.632 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:12.647 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:12.647 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:12.664 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:12.664 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:12.672 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:12.671+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:12.675 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:12.676 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:12.679 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:12.680 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:12.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:12.696 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:12.772 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:12.771+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:5 1.2 e14/12 1:5cea7035:::rbd_info:head v 14'1, mlcod=0'0)", 2024-09-17T07:47:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.801356054999999, 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067261100000000004, 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0899999999999997e-07 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996967+0000", 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996982+0000", 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4759e-05 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996992+0000", 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0370999999999999e-05 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997045+0000", 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2800000000000003e-05 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997079+0000", 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3834000000000002e-05 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997604+0000", 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052493799999999997 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997631+0000", 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6659999999999999e-05 2024-09-17T07:47:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:7 1.2 e15/12 1:5cea7035:::rbd_info:head v 15'2, mlcod=0'0)", 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.794999903000001, 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088176699999999999, 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003325+0000", 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003344+0000", 2024-09-17T07:47:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8304e-05 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003354+0000", 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0104e-05 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003407+0000", 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3743999999999999e-05 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003432+0000", 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4726e-05 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004160+0000", 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072798800000000005 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004196+0000", 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5663000000000001e-05 2024-09-17T07:47:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:12.809 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:12.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:12.873 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:12.873+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:12.952 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:12.952 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:12.952 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:12.952 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:12.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:13.004 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:13.003+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:13.062 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:13.062 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:13.063 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:13.063 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:13.063 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:13.071 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:13.075 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:13.075+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:13.177 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:13.175+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:13.215 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:13.215 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:13.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:13.215 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.213196908, 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:13.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:13.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.200423375, 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:13.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:13.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.197216263000001, 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.227 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:13.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:13.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.196745894999999, 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:13.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:13.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:13.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.196162695999998, 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:13.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.195672173999998, 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:13.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:13.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:13.245 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.193155440999998, 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.182208857999999, 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:13.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:13.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:13.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.044429662999999, 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:13.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:13.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:13.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:13.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.043971909, 2024-09-17T07:47:13.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:13.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.728968461000001, 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:13.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.726034234, 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.270 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:13.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:13.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.725581076000001, 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:13.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.725145043000001, 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:13.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.283 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.724567507, 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:13.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.724023058, 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:13.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:13.292 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.723639747, 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:13.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:13.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.720365805, 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:13.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.311020509999999, 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.301 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:13.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:13.305 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.30905521, 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:13.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:13.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:13.310 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:13.310 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:13.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:13.312 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:13.310+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:13.412 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:13.411+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:13.455 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:13.455 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:13.455 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:13.455 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:13.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:13.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:47:13.548 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:13.547+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:13.606 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:13.607 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:13.614 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:13.613+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:13.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:13.715 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:13.714+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:13.758 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:13.759 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:13.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:13.816 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:13.815+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:13.877 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:13.877 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:13.878 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:13.878 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:13.886 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:13.916 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:13.916+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:14.017 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:14.016+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:14.025 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.025 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:14.025 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:14.025 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:14.026 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:14.118 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:14.118+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972355830000001, 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.134 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:14.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972287476999998, 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:14.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:14.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972229371000001, 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.146 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:14.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972207989000001, 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.152 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:14.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.157 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972193073, 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:14.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:14.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:14.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972141424, 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:14.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:14.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972085908, 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.972030352000001, 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:14.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:14.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.97195765, 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.182 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:14.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:14.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.971942776999999, 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:14.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:14.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.193 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.643909273999999, 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:14.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:14.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.643883110000001, 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.201 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.64386227, 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.208 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.643782026, 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:14.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:14.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.643769044999999, 2024-09-17T07:47:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.643752867, 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.643686376000002, 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.230041556, 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.230027915000001, 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.229990086000001, 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.266 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:14.268 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:14.267+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:14.369 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:14.368+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:14.406 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.406 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:14.406 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:14.407 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.415 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:14.469 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:14.469+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:14.511 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.512 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:14.512 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:14.512 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:14.512 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.520 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:47:14.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:14.577 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:14.576+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:14.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.613199383000001, 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:14.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.612727035999999, 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:14.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:14.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.623054053000001, 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.628 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:14.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:14.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.635 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.620606863999999, 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:14.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.616954599, 2024-09-17T07:47:14.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:14.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:14.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.614045057999999, 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:14.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:14.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:14.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.652 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:14.652 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:14.652 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:14.672 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:14.671+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:14.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:14.751 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.752 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:14.752 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:14.752 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:14.803 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:14.803+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:14.838 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:14.838 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:14.854 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:14.854 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:14.857 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:14.857 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:14.857 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:14.857 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:14.857 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:14.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:14.869 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:14.869 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:14.885 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:14.885 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:14.901 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:14.901 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:14.906 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:14.906+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:14.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:14.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:14.975 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:14.974+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:15.003 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.004 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:15.075 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:15.074+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:15.114 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.114 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:15.176 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:15.176+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:15.264 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.264 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.273 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:15.278 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:15.278+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:15.374 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.375 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.379 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:15.377+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:15.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:15.480 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:15.479+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:15.525 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.526 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:15.577 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:15.577 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:15.577 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-17T07:47:15.577 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:47:15.578 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:47:15.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-17T07:47:15.618 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:15.617+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:15.677 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.677 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.682 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:15.681+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:15.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:15.721 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:47:15.721 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:15.782 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:15.782+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:15.790 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.790 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:15.883 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:15.882+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:15.900 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:15.900 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:15.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:15.984 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:15.984+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:16.009 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.010 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:16.085 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:16.084+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:16.119 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.119 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:16.186 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:16.185+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:16.238 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.238 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:16.291 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:16.291+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:16.344 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.344 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:16.388 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:16.386+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:16.489 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:16.489+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:16.490 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.490 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:16.537 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:47:16.590 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:16.589+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:16.637 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.637 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:16.731 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:16.731+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:16.786 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.786 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.791 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:16.790+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:16.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:16.892 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:16.892+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:16.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:16.932 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:16.932 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:16.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:17.024 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:17.024+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:17.045 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:17.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:17.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:17.079 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:17.080 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:17.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:17.092 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:17.092 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:17.108 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:17.108 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:17.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:17.129 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:17.129+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:17.194 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:17.193+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:17.225 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:17.226 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:17.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:17.295 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:17.295+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:17.331 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:17.331 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:17.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:17.331 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:17.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 24.330422707, 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:17.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:17.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:17.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:17.336 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:17.336 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:17.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:17.397 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:17.395+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:17.444 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:17.444 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:17.444 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:17.444 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:17.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:17.497 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:17.497+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:17.531 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:47:17.596 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:17.597 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:17.597 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:17.597 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:17.597 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:17.598 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:17.597+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:17.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:17.699 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:17.698+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:5 1.2 e14/12 1:5cea7035:::rbd_info:head v 14'1, mlcod=0'0)", 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.751106518, 2024-09-17T07:47:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067261100000000004, 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0899999999999997e-07 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996967+0000", 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996982+0000", 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4759e-05 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996992+0000", 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0370999999999999e-05 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997045+0000", 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2800000000000003e-05 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997079+0000", 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3834000000000002e-05 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997604+0000", 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052493799999999997 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997631+0000", 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6659999999999999e-05 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:7 1.2 e15/12 1:5cea7035:::rbd_info:head v 15'2, mlcod=0'0)", 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.744750366000002, 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088176699999999999, 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-09-17T07:47:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003325+0000", 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003344+0000", 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8304e-05 2024-09-17T07:47:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003354+0000", 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0104e-05 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003407+0000", 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3743999999999999e-05 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003432+0000", 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4726e-05 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004160+0000", 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072798800000000005 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004196+0000", 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5663000000000001e-05 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:17.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:17.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:17.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:17.758 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:17.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:17.801 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:17.799+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:17.901 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:17.902 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:17.902 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:17.900+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:17.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:18.002 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:18.001+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:18.051 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:18.052 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:18.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:18.104 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:18.102+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:18.203 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.201825540000002, 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:18.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:18.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:18.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.189052007000001, 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:18.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.185844894999999, 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.214 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:18.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:18.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:18.218 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.185374527, 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:18.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.184791327999999, 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.184300806, 2024-09-17T07:47:18.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:18.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:18.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.181784072999999, 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:18.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:18.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.17083749, 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:18.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:18.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.033058295, 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.241 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:18.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:18.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.032600541000001, 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.246 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:18.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:18.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.717597092999998, 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:18.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:18.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.714662866000001, 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.255 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.714209707999998, 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:18.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.713773674999999, 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:18.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.713196139000001, 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.712651690000001, 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.272 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:18.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:18.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.712268379000001, 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:18.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.708994437000001, 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.281 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:18.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.299649142, 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:18.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.297683842000001, 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:18.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:18.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:18.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:18.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:18.295 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:18.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:18.297 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:18.296+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:18.397 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:18.395+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:18.399 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:18.399 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:18.399 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:18.399 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:18.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:18.498 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:18.496+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:18.549 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:18.550 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:18.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:18.599 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:18.598+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:18.699 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:18.698+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:18.700 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:18.700 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:18.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:18.800 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:18.800+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:18.818 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:18.818 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:18.818 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:18.818 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:18.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:18.911 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:18.911+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:18.965 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:18.965 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:18.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:19.016 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:19.016+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:19.104 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:19.104+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953982438000001, 2024-09-17T07:47:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953914085000001, 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953855979, 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953834597, 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953819680999999, 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953768031999999, 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953712516, 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.95365696, 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953584257999999, 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:19.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:19.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.953569385000002, 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:19.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:19.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:19.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625535882000001, 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625509718, 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:19.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:19.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625488877999999, 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.191 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:19.196 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625408633999999, 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:19.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625395652999998, 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:19.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:19.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625379474999999, 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:19.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.625312984000001, 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.211668163999999, 2024-09-17T07:47:19.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.211654523, 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.211616694, 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:19.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:19.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:19.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:19.242 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:19.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:19.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:19.245 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:19.244+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:19.345 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:19.345+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:19.368 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:19.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:19.383 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:19.383 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:19.383 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:19.383 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:19.383 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:19.384 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:19.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:19.398 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:19.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:19.414 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:19.429 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:19.445 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:19.476 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:19.476+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:19.529 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:19.529 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:19.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:19.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:47:19.579 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:19.579+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:19.647 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:19.646+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:19.678 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:19.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:19.748 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:19.748+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:19.784 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:19.784 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:19.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:19.849 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:19.848+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:19.929 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:19.929 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:19.938 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:19.951 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:19.950+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:20.037 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.037 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:20.090 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:20.089+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:20.148 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.149 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.152 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:20.150+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:20.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:20.253 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:20.252+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:20.299 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.300 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.308 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:20.353 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:20.353+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:20.409 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.409 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:20.455 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:20.455+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:20.519 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.519 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:20.534 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:47:20.556 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:20.555+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:20.632 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.633 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.639 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:20.683 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:20.682+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:20.731 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:20.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:20.731 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.3944685039193816 2024-09-17T07:47:20.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 0.3944685039193816 2024-09-17T07:47:20.741 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.741 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.750 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:20.758 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:20.758+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:20.859 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:20.859+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:20.894 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:20.895 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:20.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:20.960 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:20.960+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:21.005 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:21.005 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:21.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:21.061 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:21.060+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:21.116 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:21.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:21.165 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:21.165+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:21.233 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:21.233 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:21.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:21.286 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:21.286+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:21.341 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:21.341 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:21.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:21.364 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:21.364+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:21.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:21.465 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:21.465+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:21.487 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:21.487 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:21.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:21.542 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 0.394469 (64fb) 2024-09-17T07:47:21.566 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:21.565+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:21.580 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:21.580 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:21.596 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:21.596 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:21.611 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:21.611 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:21.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:21.627 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:21.634 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:21.634 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:21.642 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:21.642 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:21.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:21.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:21.657 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:21.666 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:21.666+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:21.741 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:21.741 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:21.750 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:21.767 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:21.767+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.841672168999999, 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.841199822, 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.851526839000002, 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.84907965, 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.845427385000001, 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.842517844, 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:21.882 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:21.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:21.884 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:21.882+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:21.981 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:21.981 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:21.981 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:21.981 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:21.984 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:21.983+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:21.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:22.085 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:22.084+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:22.087 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:22.087 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:22.087 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:22.087 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:22.087 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:22.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:22.186 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:22.186+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:22.233 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.232637988, 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:22.239 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:22.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:22.287 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:22.287+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:22.357 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:22.357 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:22.357 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:22.357 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:22.366 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:22.387 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:22.387+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:22.468 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:22.469 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:22.477 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:22.520 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:22.520+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:22.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:47:22.580 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:5 1.2 e14/12 1:5cea7035:::rbd_info:head v 14'1, mlcod=0'0)", 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.58396012, 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067261100000000004, 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.581 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0899999999999997e-07 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996967+0000", 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996982+0000", 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4759e-05 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996992+0000", 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0370999999999999e-05 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997045+0000", 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2800000000000003e-05 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997079+0000", 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3834000000000002e-05 2024-09-17T07:47:22.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997604+0000", 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052493799999999997 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997631+0000", 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6659999999999999e-05 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:7 1.2 e15/12 1:5cea7035:::rbd_info:head v 15'2, mlcod=0'0)", 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.577603967999998, 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088176699999999999, 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003325+0000", 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:47:22.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003344+0000", 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8304e-05 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003354+0000", 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0104e-05 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003407+0000", 2024-09-17T07:47:22.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3743999999999999e-05 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003432+0000", 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4726e-05 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004160+0000", 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072798800000000005 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004196+0000", 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5663000000000001e-05 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.590 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:22.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:22.592 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:22.591+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:22.692 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:22.692+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:22.735 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:22.735 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:22.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:22.793 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:22.793+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:22.846 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:22.846 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:22.846 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:22.846 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:22.846 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:22.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:22.895 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:22.894+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:22.956 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.954838586000001, 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.957 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:22.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.942065053, 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.963 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:22.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:22.967 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.938857940999998, 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:22.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:22.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.938387573, 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:22.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:22.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.937804373999999, 2024-09-17T07:47:22.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:22.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:22.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.937313851999999, 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:22.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:22.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.934797118999999, 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.923850536, 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:22.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.786071341, 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:22.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.785613587, 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:22.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.470610139000001, 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:23.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.007 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.467675912000001, 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.467222754000002, 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.466786720999998, 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.466209185, 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.465664736000001, 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.465281425000001, 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.462007483000001, 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.052662187999999, 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.050696888000001, 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:23.047 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:23.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:23.049 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:23.048+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:23.149 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:23.149+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:23.192 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:23.192 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:23.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:23.284 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:23.284+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:23.342 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:23.343 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:23.343 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:23.343 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:23.343 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:23.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:23.352 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:23.351+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:23.452 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:23.451+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:23.493 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:23.494 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:23.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:23.494 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:23.494 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:23.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:23.549 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:47:23.553 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:23.553+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:23.612 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:23.612 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:23.612 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:23.612 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:23.621 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:23.664 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:23.664+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:23.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:23.719 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:23.719 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:23.719 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:23.728 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:23.755 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:23.755+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:23.785 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:23.801 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:23.816 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:23.832 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:23.832 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:23.848 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:23.849 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:23.856 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:23.856+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:23.864 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:23.864 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:23.868 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:23.868 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:23.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:23.961 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:23.961+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:24.015 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.015 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.024 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:24.066 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:24.065+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:24.159 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:24.159+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:24.162 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.162 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:24.260 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:24.260+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:24.267 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.268 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:24.362 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:24.361+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:24.415 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.415 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:24.468 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:24.468+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:24.522 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.522 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:24.571 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:24.571+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:24.665 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:24.664+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:24.668 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.668 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:24.766 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:24.765+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:24.792 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.792 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:24.885 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:24.885+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:24.945 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:24.945 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:24.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:24.967 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:24.967+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:25.068 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:25.068+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:25.095 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.095 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:25.169 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:25.169+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:25.205 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.206 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:25.270 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:25.269+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:25.315 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.315 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:25.371 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:25.371+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:25.425 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.425 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:25.477 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:25.477+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:25.535 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.535 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:25.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:47:25.585 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:25.584+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:25.674 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:25.674+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:25.687 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.687 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:25.782 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:25.782+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:25.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:25.895 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:25.895+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:25.959 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:25.959 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:25.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:25.978 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:25.978+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:26.005 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:26.005 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:26.021 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:26.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:26.051 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:26.051 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:26.066 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:26.067 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:26.080 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:26.079+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:26.083 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:26.083 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:26.108 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:26.108 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:26.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:26.180 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:26.180+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:26.256 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:26.256 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095306145999999, 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:26.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095237793000003, 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095179686999998, 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:26.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:26.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:26.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:26.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:26.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:26.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095158304999998, 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.278 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:26.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:26.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:26.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.284 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095143389, 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.285 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:26.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:26.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:26.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095091740000001, 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.292 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:26.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.297 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.095036223999998, 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.303 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.094980667999998, 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:26.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:26.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:26.309 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.094907966000001, 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:26.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.094893093000003, 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.315 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:26.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:26.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766859590000003, 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:26.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:26.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766833425999998, 2024-09-17T07:47:26.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:26.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:26.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:26.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766812586, 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:26.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766732341999997, 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.339 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:26.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766719361, 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:26.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:26.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766703182999997, 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:26.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:26.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.766636691999999, 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:26.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.352991871999997, 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:26.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:26.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.352978231000002, 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:26.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:26.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.352940402000002, 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:26.374 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:26.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:26.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:26.380 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.380 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.380 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.380 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.380 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.380 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:26.381 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:26.381+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:26.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:26.480 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:26.480 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:26.480 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:26.481 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:26.481 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:26.481+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:26.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:47:26.555 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:26.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:26.555 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-17T07:47:26.555 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:47:26.556 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:47:26.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-17T07:47:26.582 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:26.582+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:26.628 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:26.628 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:26.628 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:26.628 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:26.628 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:26.636 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:26.664 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:47:26.664 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:26.684 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:26.683+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:26.775 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:26.775 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:26.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.769801751999999, 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:26.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 33.769329405000001, 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:26.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.779656422000002, 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:26.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:26.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:26.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:26.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:26.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:26.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.777209233000001, 2024-09-17T07:47:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.773556968000001, 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.770647427, 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:26.811 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:26.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:26.814 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:26.812+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:26.912 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:26.912 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:26.912 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:26.912 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:26.913 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:26.913+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:26.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:27.013 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:27.013+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:27.018 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:27.018 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:27.018 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:27.018 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:27.018 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:27.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:27.114 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:27.114+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.164960737000001, 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:27.170 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:27.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:27.219 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:27.219+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:27.289 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:27.289 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:27.289 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:27.289 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:27.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:27.340 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:27.339+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:27.418 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:27.417+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:27.442 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:27.443 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:27.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:27.536 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:27.536+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:27.593 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:5 1.2 e14/12 1:5cea7035:::rbd_info:head v 14'1, mlcod=0'0)", 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.596851104999999, 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067261100000000004, 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:27.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0899999999999997e-07 2024-09-17T07:47:27.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996967+0000", 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996982+0000", 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4759e-05 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:27.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996992+0000", 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0370999999999999e-05 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997045+0000", 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2800000000000003e-05 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997079+0000", 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3834000000000002e-05 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997604+0000", 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052493799999999997 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997631+0000", 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6659999999999999e-05 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:7 1.2 e15/12 1:5cea7035:::rbd_info:head v 15'2, mlcod=0'0)", 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.590494952999997, 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088176699999999999, 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003325+0000", 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003344+0000", 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8304e-05 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003354+0000", 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0104e-05 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003407+0000", 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3743999999999999e-05 2024-09-17T07:47:27.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003432+0000", 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4726e-05 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004160+0000", 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072798800000000005 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004196+0000", 2024-09-17T07:47:27.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5663000000000001e-05 2024-09-17T07:47:27.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:27.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:27.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:27.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:27.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:27.603 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:27.604 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:27.645 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:27.645+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:27.719 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:27.719+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:27.746 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:27.747 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:27.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:27.821 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:27.820+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:27.856 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:27.857 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:27.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:27.949 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:27.949+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:28.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.007310312999998, 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.994536779999997, 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.991329667999999, 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:28.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.990859299999997, 2024-09-17T07:47:28.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:28.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.990276100999999, 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.989785578999999, 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.987268845999999, 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.976322263, 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:28.047 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.838543068, 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.838085313999997, 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.523081865999998, 2024-09-17T07:47:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:28.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.520147639000001, 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.519694481000002, 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.519258448000002, 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:28.074 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.518680912000001, 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:28.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:28.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:28.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.518136462999998, 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:28.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.517753151999997, 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:28.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.514479209999998, 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.105133915000003, 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.103168615000001, 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:28.102 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:28.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:28.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:28.105 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:28.103+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:28.205 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:28.204+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:28.250 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:28.250 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:28.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:28.263 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:28.263 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:28.279 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:28.280 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:28.294 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:28.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:28.306 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:28.306+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:28.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:28.311 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:28.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:28.327 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:28.343 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:28.365 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:28.365 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:28.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:28.414 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:28.414+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:28.508 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:28.508+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:28.515 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:28.515 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:28.523 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:28.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:47:28.609 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:28.609+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:28.634 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:28.634 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:28.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:28.727 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:28.727+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:28.781 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:28.782 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:28.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:28.831 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:28.831+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:28.912 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:28.912+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:28.927 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:28.927 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:28.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:29.020 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:29.019+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:29.073 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.073 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:29.122 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:29.122+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:29.215 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:29.215+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:29.218 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.218 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:29.316 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:29.316+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:29.365 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.365 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:29.417 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:29.417+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:29.471 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.471 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:29.523 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:29.523+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:29.563 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:47:29.577 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.577 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:29.626 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:29.626+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:29.720 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:29.720+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:29.723 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.723 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:29.821 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:29.821+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:29.844 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.844 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:29.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:29.922 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:29.922+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:29.997 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:29.997 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:30.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:30.023 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:30.023+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:30.106 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:30.107 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:30.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:30.124 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:30.124+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:30.216 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:30.216 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:30.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:30.225 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:30.225+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:30.326 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:30.326 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:30.327 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:30.326+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:30.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:30.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:30.427 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:30.426+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:30.479 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:30.479 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:30.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:30.493 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:30.509 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:30.510 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:30.525 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:30.527 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:30.527+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:30.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:30.557 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:30.557 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:30.574 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:30.597 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:30.597 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:30.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:30.628 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:30.628+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:30.729 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:30.729+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:30.747 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:30.747 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:30.748 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:30.748 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:30.748 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:30.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:30.830 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:30.830+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:30.857 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:30.857 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:30.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:30.931 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:30.931+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:30.975 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:30.976 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:30.976 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:30.976 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:30.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:31.032 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:31.032+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:31.082 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.082 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:31.083 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:31.083 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:31.083 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:31.133 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:31.133+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:31.189 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028403048000001, 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:31.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:31.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:31.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:31.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:31.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.196 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028334694999998, 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:31.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:31.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028276589000001, 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.203 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028255207000001, 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:31.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:31.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028240291000003, 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:31.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:31.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:31.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:31.221 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028188642000003, 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:31.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:31.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.227 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028133126, 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028077570000001, 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:31.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:31.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:31.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.028004867999996, 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:31.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.244 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.027989994999999, 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:31.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699956491999998, 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.250 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:31.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:31.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699930328000001, 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:31.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:31.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.261 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699909488000003, 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.262 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699829244, 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.268 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:31.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:31.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:31.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.274 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699816263000002, 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:31.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:31.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:31.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:31.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699800085, 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.286 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.699733594000001, 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:31.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:31.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:31.292 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.286088774, 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.293 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:31.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.298 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.286075132999997, 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:31.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.304 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.286037303999997, 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:31.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.310 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.311 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.312 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:31.311+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:31.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:31.410 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:31.410 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.412 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:31.412+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:31.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:31.513 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:31.513+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:31.516 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.516 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:31.516 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:31.516 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:31.516 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:31.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:47:31.614 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:31.614+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.659330079, 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.658857732000001, 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.669184749000003, 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.666737560000001, 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.663085295000002, 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.660175754000001, 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:31.697 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.698 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:31.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:31.698 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T07:47:31.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-17T07:47:31.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:31.741 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:31.741+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:31.815 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:31.815+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:31.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-17T07:47:31.839 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.839 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:31.839 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:31.839 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.848 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:31.916 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:31.916+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:31.945 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:31.946 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:31.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:32.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-17T07:47:32.017 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:32.016+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:32.051 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 39.050842490000001, 2024-09-17T07:47:32.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:32.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:32.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:32.057 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:32.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:32.118 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:32.118+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:32.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-17T07:47:32.188 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:32.188 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:32.188 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:32.188 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:32.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:32.218 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:32.219+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:32.305 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-17T07:47:32.320 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:32.319+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:32.342 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:32.343 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:32.343 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:32.343 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:32.343 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:32.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:32.421 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:32.421+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:32.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-17T07:47:32.496 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:5 1.2 e14/12 1:5cea7035:::rbd_info:head v 14'1, mlcod=0'0)", 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 38.499950204999998, 2024-09-17T07:47:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067261100000000004, 2024-09-17T07:47:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0899999999999997e-07 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996967+0000", 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996982+0000", 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4759e-05 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996992+0000", 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0370999999999999e-05 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997045+0000", 2024-09-17T07:47:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2800000000000003e-05 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997079+0000", 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3834000000000002e-05 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997604+0000", 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052493799999999997 2024-09-17T07:47:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997631+0000", 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6659999999999999e-05 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:7 1.2 e15/12 1:5cea7035:::rbd_info:head v 15'2, mlcod=0'0)", 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.493594053000002, 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088176699999999999, 2024-09-17T07:47:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003325+0000", 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:47:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003344+0000", 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8304e-05 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003354+0000", 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0104e-05 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003407+0000", 2024-09-17T07:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3743999999999999e-05 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003432+0000", 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4726e-05 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004160+0000", 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072798800000000005 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004196+0000", 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5663000000000001e-05 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:32.509 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:32.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:32.522 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:32.522+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:32.564 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:47:32.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:32.616 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:32.617 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:32.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:32.625 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:32.624+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:32.723 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:32.723+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:32.736 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:32.736 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:32.752 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:32.752 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:32.783 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:32.786 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:32.786 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:32.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:32.806 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:32.822 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:32.824 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:32.823+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:32.838 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:32.838 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:32.909 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:32.909 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:32.917 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:32.961 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:32.961+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:33.019 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.020 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.025 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:33.024+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:33.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:33.125 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:33.125+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:33.170 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.170 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:33.227 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:33.227+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:33.280 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.280 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:33.332 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:33.332+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:33.386 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.387 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:33.437 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:33.436+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:33.529 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:33.528+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:33.533 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.533 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.542 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:33.630 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:33.629+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:33.679 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.680 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:33.733 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:33.730+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:33.785 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.785 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:33.837 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:33.837+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:33.891 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:33.891 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:33.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:33.940 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:33.940+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:34.034 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:34.034+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:34.037 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.037 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:34.135 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:34.135+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:34.143 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.143 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:34.236 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:34.236+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:34.249 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.250 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:34.337 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:34.337+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:34.396 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.397 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.405 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:34.438 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:34.438+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:34.520 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.520 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:34.539 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:34.539+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:34.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:47:34.633 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.633 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.640 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:34.640+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:34.642 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:34.741 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:34.741+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:34.750 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.750 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:34.842 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:34.842+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:34.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:34.904 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:34.904 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:34.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:34.956 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:34.956+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:34.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:34.970 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:34.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:35.000 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:35.016 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:35.017 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:35.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:35.017 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:35.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:35.033 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:35.033 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:35.048 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:35.049 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:35.049 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:35.048+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.134201758000003, 2024-09-17T07:47:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.121428225000002, 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.118221112999997, 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.117750745000002, 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.117167545999997, 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.116677023999998, 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.114160290999997, 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.103213707999998, 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.965434512999998, 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.964976759000002, 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.649973310999997, 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.647039083999999, 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.646585926, 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.646149893, 2024-09-17T07:47:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.645572356999999, 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.645027908000003, 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.644644597000003, 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.641370655000003, 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:35.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:35.218 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.232025360000002, 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:35.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.23006006, 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:35.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:35.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:35.227 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:35.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:35.229 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:35.228+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:35.329 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:35.329+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:35.371 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:35.371 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:35.371 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:35.371 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:35.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:35.463 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:35.463+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:35.521 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:35.521 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:35.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:35.571 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:35.570+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:35.575 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:47:35.632 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:35.632+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:35.673 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:35.673 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:35.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:35.733 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:35.733+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:35.791 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:35.791 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:35.791 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:35.791 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:35.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:35.834 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:35.833+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:35.898 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:35.898 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:35.907 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:35.950 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:35.950+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844647395000003, 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.006 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:36.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844579041999999, 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:36.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:36.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844520936000002, 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:36.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:36.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:36.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:36.026 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844499554000002, 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.032 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844484637999997, 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:36.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:36.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844432988999998, 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:36.040 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:36.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844377473000002, 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.047 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:36.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:36.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:36.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:36.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:36.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:36.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:36.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:36.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:36.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.056 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844321917000002, 2024-09-17T07:47:36.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:36.058 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.058 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:36.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:36.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:36.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:36.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:36.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:36.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:36.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:36.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:36.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:36.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:36.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:36.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:36.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:36.069 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844249214999998, 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:36.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:36.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:36.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:36.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.076 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.844234342, 2024-09-17T07:47:36.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.079 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:36.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:36.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:36.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:36.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:36.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:36.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:36.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.086 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.516200839, 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:36.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:36.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.093 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.516174675000002, 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.094 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:36.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:36.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.516153834999997, 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.516073591000001, 2024-09-17T07:47:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:36.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:36.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.123 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.123 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:36.123 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.516060609999997, 2024-09-17T07:47:36.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:36.124 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.124 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.125 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:36.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.125 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:36.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:36.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:36.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:36.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.516044432000001, 2024-09-17T07:47:36.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:36.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.515977941000003, 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:36.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.102333121000001, 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.143 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:36.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.148 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.102319479999998, 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:36.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:36.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.102281650999998, 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:36.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:36.159 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.160 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:36.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:36.162 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:36.161+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:36.262 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:36.262+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:36.301 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:36.301 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:36.301 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:36.301 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:36.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:36.363 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:36.363+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:36.448 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:36.448 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:36.448 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:36.448 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:36.448 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:36.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:36.464 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:36.463+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:36.554 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.549267153999999, 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:36.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.548794807, 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.559 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:36.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.559121824000002, 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:36.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:36.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:36.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:36.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.569 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.556674635, 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:36.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.573 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.553022370000001, 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:36.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:36.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:36.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:36.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:36.580 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.550112829, 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:36.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:36.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:36.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:36.585 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:36.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:36.588 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:36.586+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:36.687 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:36.686+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:36.724 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:36.724 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:36.724 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:36.725 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:36.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:36.817 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:36.817+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:36.871 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:36.871 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:36.871 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:36.871 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:36.871 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:36.879 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:36.889 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:36.888+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:36.990 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:36.990+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:37.017 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.016621690000001, 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:37.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:37.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:37.022 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:37.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:37.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:37.113 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:37.113+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:37.142 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:37.142 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:37.142 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:37.142 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:37.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:37.191 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:37.191+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:37.194 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:37.195 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:37.210 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:37.210 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:37.226 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:37.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:37.241 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:37.241 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:37.257 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:37.257 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:37.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:37.273 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:37.292 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:37.292+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:37.297 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:37.298 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:37.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:37.393 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:37.393+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:37.448 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:37.448 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:37.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:37.498 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:37.498+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:37.574 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:37.574 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:37.575 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T07:47:37.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:47:37.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:47:37.595 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:37.595+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:37.601 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:37.601 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:37.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:37.696 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:37.696+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:37.758 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:37.758 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:37.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:37.812 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:37.812+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 35, 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:47:35.570272+0000", 2024-09-17T07:47:37.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:46:50.986897+0000", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:46:43.251109+0000", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5801984, 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 4, 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:47:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 3, 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:47:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:47:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "20", 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:47:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 18, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:47:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:47:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:47:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:47:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:47:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:47:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:47:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:47:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:47:37.870 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:47:37.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "24", 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 18, 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:47:37.872 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:47:37.873 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:47:37.874 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:47:37.875 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:47:37.876 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 29, 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/31884", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/31884", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/31884", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/31884", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:47:37.881 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 29, 2024-09-17T07:47:37.882 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:47:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:47:37.888 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.893 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.3944549560546875, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 12, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/31604", 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/31604", 2024-09-17T07:47:37.898 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/31604", 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/31604", 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:47:37.899 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 16, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:47:37.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:47:37.904 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:47:37.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 12, 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/31606", 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/31606", 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/31606", 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/31606", 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:47:37.910 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:47:37.911 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:47:37.913 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:47:37.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:47:37.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-09-17T07:47:37.916 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:37.917 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:37.918 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:37.919+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:37.920 INFO:tasks.thrashosds.thrasher:j is {'epoch': 35, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:47:35.570272+0000', 'last_up_change': '2024-09-17T07:46:50.986897+0000', 'last_in_change': '2024-09-17T07:46:43.251109+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 31884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 31884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 31884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 31884}]}, 'public_addr': '172.21.15.26:6801/31884', 'cluster_addr': '172.21.15.26:6803/31884', 'heartbeat_back_addr': '172.21.15.26:6807/31884', 'heartbeat_front_addr': '172.21.15.26:6805/31884', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 1, 'in': 1, 'weight': 0.3944549560546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 12, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 31604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 31604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 31604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 31604}]}, 'public_addr': '172.21.15.94:6801/31604', 'cluster_addr': '172.21.15.94:6803/31604', 'heartbeat_back_addr': '172.21.15.94:6807/31604', 'heartbeat_front_addr': '172.21.15.94:6805/31604', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 12, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 31606}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 31606}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 31606}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 31606}]}, 'public_addr': '172.21.15.94:6817/31606', 'cluster_addr': '172.21.15.94:6819/31606', 'heartbeat_back_addr': '172.21.15.94:6823/31606', 'heartbeat_front_addr': '172.21.15.94:6821/31606', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:47:37.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:47:37.921 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:37.921 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:37.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:38.020 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:38.019+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:38.064 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.064 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:38.121 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:38.121+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:38.216 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.216 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.222 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:38.222+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:38.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:38.234 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:38.234 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:47:38.247 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-17T07:47:37.416718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1812532,"kb_used_data":3436,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564418508,"statfs":{"total":579820584960,"available":577964552192,"internally_reserved":0,"allocated":3518464,"data_stored":2041849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.854897"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627495+0000","last_change":"2024-09-17T07:46:55.313728+0000","last_active":"2024-09-17T07:47:32.627495+0000","last_peered":"2024-09-17T07:47:32.627495+0000","last_clean":"2024-09-17T07:47:32.627495+0000","last_became_active":"2024-09-17T07:46:53.114488+0000","last_became_peered":"2024-09-17T07:46:53.114488+0000","last_unstale":"2024-09-17T07:47:32.627495+0000","last_undegraded":"2024-09-17T07:47:32.627495+0000","last_fullsized":"2024-09-17T07:47:32.627495+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:03.738226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.564229+0000","last_change":"2024-09-17T07:47:23.321116+0000","last_active":"2024-09-17T07:47:32.564229+0000","last_peered":"2024-09-17T07:47:32.564229+0000","last_clean":"2024-09-17T07:47:32.564229+0000","last_became_active":"2024-09-17T07:47:23.320796+0000","last_became_peered":"2024-09-17T07:47:23.320796+0000","last_unstale":"2024-09-17T07:47:32.564229+0000","last_undegraded":"2024-09-17T07:47:32.564229+0000","last_fullsized":"2024-09-17T07:47:32.564229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:21:36.725627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627678+0000","last_change":"2024-09-17T07:46:55.003172+0000","last_active":"2024-09-17T07:47:32.627678+0000","last_peered":"2024-09-17T07:47:32.627678+0000","last_clean":"2024-09-17T07:47:32.627678+0000","last_became_active":"2024-09-17T07:46:53.114521+0000","last_became_peered":"2024-09-17T07:46:53.114521+0000","last_unstale":"2024-09-17T07:47:32.627678+0000","last_undegraded":"2024-09-17T07:47:32.627678+0000","last_fullsized":"2024-09-17T07:47:32.627678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:46.450177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037727799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":42,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627597+0000","last_change":"2024-09-17T07:47:23.428235+0000","last_active":"2024-09-17T07:47:32.627597+0000","last_peered":"2024-09-17T07:47:32.627597+0000","last_clean":"2024-09-17T07:47:32.627597+0000","last_became_active":"2024-09-17T07:47:23.427998+0000","last_became_peered":"2024-09-17T07:47:23.427998+0000","last_unstale":"2024-09-17T07:47:32.627597+0000","last_undegraded":"2024-09-17T07:47:32.627597+0000","last_fullsized":"2024-09-17T07:47:32.627597+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.590519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.563944+0000","last_change":"2024-09-17T07:47:01.487813+0000","last_active":"2024-09-17T07:47:32.563944+0000","last_peered":"2024-09-17T07:47:32.563944+0000","last_clean":"2024-09-17T07:47:32.563944+0000","last_became_active":"2024-09-17T07:47:01.487515+0000","last_became_peered":"2024-09-17T07:47:01.487515+0000","last_unstale":"2024-09-17T07:47:32.563944+0000","last_undegraded":"2024-09-17T07:47:32.563944+0000","last_fullsized":"2024-09-17T07:47:32.563944+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:46:36.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":120,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.563629+0000","last_change":"2024-09-17T07:46:52.996274+0000","last_active":"2024-09-17T07:47:32.563629+0000","last_peered":"2024-09-17T07:47:32.563629+0000","last_clean":"2024-09-17T07:47:32.563629+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:47:32.563629+0000","last_undegraded":"2024-09-17T07:47:32.563629+0000","last_fullsized":"2024-09-17T07:47:32.563629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:03.825313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627603+0000","last_change":"2024-09-17T07:46:56.024711+0000","last_active":"2024-09-17T07:47:32.627603+0000","last_peered":"2024-09-17T07:47:32.627603+0000","last_clean":"2024-09-17T07:47:32.627603+0000","last_became_active":"2024-09-17T07:46:52.998238+0000","last_became_peered":"2024-09-17T07:46:52.998238+0000","last_unstale":"2024-09-17T07:47:32.627603+0000","last_undegraded":"2024-09-17T07:47:32.627603+0000","last_fullsized":"2024-09-17T07:47:32.627603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:52.906241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627637+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:47:32.627637+0000","last_peered":"2024-09-17T07:47:32.627637+0000","last_clean":"2024-09-17T07:47:32.627637+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:47:32.627637+0000","last_undegraded":"2024-09-17T07:47:32.627637+0000","last_fullsized":"2024-09-17T07:47:32.627637+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:05:30.296358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":36,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627683+0000","last_change":"2024-09-17T07:46:55.003565+0000","last_active":"2024-09-17T07:47:32.627683+0000","last_peered":"2024-09-17T07:47:32.627683+0000","last_clean":"2024-09-17T07:47:32.627683+0000","last_became_active":"2024-09-17T07:46:52.999808+0000","last_became_peered":"2024-09-17T07:46:52.999808+0000","last_unstale":"2024-09-17T07:47:32.627683+0000","last_undegraded":"2024-09-17T07:47:32.627683+0000","last_fullsized":"2024-09-17T07:47:32.627683+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:09.551569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627678+0000","last_change":"2024-09-17T07:47:01.488782+0000","last_active":"2024-09-17T07:47:32.627678+0000","last_peered":"2024-09-17T07:47:32.627678+0000","last_clean":"2024-09-17T07:47:32.627678+0000","last_became_active":"2024-09-17T07:47:01.488528+0000","last_became_peered":"2024-09-17T07:47:01.488528+0000","last_unstale":"2024-09-17T07:47:32.627678+0000","last_undegraded":"2024-09-17T07:47:32.627678+0000","last_fullsized":"2024-09-17T07:47:32.627678+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:08.762766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627471+0000","last_change":"2024-09-17T07:46:55.313717+0000","last_active":"2024-09-17T07:47:32.627471+0000","last_peered":"2024-09-17T07:47:32.627471+0000","last_clean":"2024-09-17T07:47:32.627471+0000","last_became_active":"2024-09-17T07:46:52.998992+0000","last_became_peered":"2024-09-17T07:46:52.998992+0000","last_unstale":"2024-09-17T07:47:32.627471+0000","last_undegraded":"2024-09-17T07:47:32.627471+0000","last_fullsized":"2024-09-17T07:47:32.627471+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:06:45.637243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.627596+0000","last_change":"2024-09-17T07:47:02.509843+0000","last_active":"2024-09-17T07:47:32.627596+0000","last_peered":"2024-09-17T07:47:32.627596+0000","last_clean":"2024-09-17T07:47:32.627596+0000","last_became_active":"2024-09-17T07:47:02.509516+0000","last_became_peered":"2024-09-17T07:47:02.509516+0000","last_unstale":"2024-09-17T07:47:32.627596+0000","last_undegraded":"2024-09-17T07:47:32.627596+0000","last_fullsized":"2024-09-17T07:47:32.627596+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:42.399749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-17T07:47:32.563627+0000","last_change":"2024-09-17T07:46:55.000486+0000","last_active":"2024-09-17T07:47:32.563627+0000","last_peered":"2024-09-17T07:47:32.563627+0000","last_clean":"2024-09-17T07:47:32.563627+0000","last_became_active":"2024-09-17T07:46:52.996744+0000","last_became_peered":"2024-09-17T07:46:52.996744+0000","last_unstale":"2024-09-17T07:47:32.563627+0000","last_undegraded":"2024-09-17T07:47:32.563627+0000","last_fullsized":"2024-09-17T07:47:32.563627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:55:26.516101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301888,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069952,"statfs":{"total":96636764160,"available":96327630848,"internally_reserved":0,"allocated":380928,"data_stored":141051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":385024,"data_stored":141070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302480,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069360,"statfs":{"total":96636764160,"available":96327024640,"internally_reserved":0,"allocated":987136,"data_stored":736342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301888,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069952,"statfs":{"total":96636764160,"available":96327630848,"internally_reserved":0,"allocated":380928,"data_stored":141051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302480,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069360,"statfs":{"total":96636764160,"available":96327024640,"internally_reserved":0,"allocated":987136,"data_stored":736342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301904,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069936,"statfs":{"total":96636764160,"available":96327614464,"internally_reserved":0,"allocated":397312,"data_stored":145993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:47:38.248 INFO:tasks.thrashosds.thrasher:Setting 1.5 to [1, 0] 2024-09-17T07:47:38.248 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.5', '1', '0'] 2024-09-17T07:47:38.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.5 1 0 2024-09-17T07:47:38.323 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:38.323+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:38.327 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.327 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:38.424 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:38.424+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:38.441 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.441 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:38.533 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:38.533+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:38.581 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:47:38.581 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.5 pg_upmap mapping to [1,0] 2024-09-17T07:47:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:38.638 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:38.638+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:38.727 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:38.727+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:38.828 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:38.828+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:38.882 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.882 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.891 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:38.934 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:38.934+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:38.988 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:38.988 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:38.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:39.030 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:39.029+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:39.131 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:39.131+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:39.134 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:39.134 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:39.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:39.232 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:39.232+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:39.239 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:39.240 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:39.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:39.284 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:39.333 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:39.332+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:39.387 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:39.387 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:39.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:39.414 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:39.414 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:39.430 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:39.434 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:39.433+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:39.445 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:39.446 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:39.461 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:39.461 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:39.476 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:39.476 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:39.492 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:39.492 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:39.493 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.492957339, 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:39.498 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:39.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:39.543 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:39.543+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:39.635 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:39.635+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:39.645 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:39.645 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:39.645 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:39.645 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:39.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:39.737 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:39.737+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:39.756 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:39.756 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:39.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:39.838 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:39.837+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:5 1.2 e14/12 1:5cea7035:::rbd_info:head v 14'1, mlcod=0'0)", 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.868955389, 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067261100000000004, 2024-09-17T07:47:39.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996958+0000", 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0899999999999997e-07 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996967+0000", 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996982+0000", 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4759e-05 2024-09-17T07:47:39.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.996992+0000", 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0370999999999999e-05 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997045+0000", 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2800000000000003e-05 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:39.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997079+0000", 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3834000000000002e-05 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997604+0000", 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052493799999999997 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.997631+0000", 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6659999999999999e-05 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:39.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4187.0:7 1.2 e15/12 1:5cea7035:::rbd_info:head v 15'2, mlcod=0'0)", 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.862599236999998, 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088176699999999999, 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:39.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003314+0000", 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003325+0000", 2024-09-17T07:47:39.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003344+0000", 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8304e-05 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003354+0000", 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0104e-05 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003407+0000", 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3743999999999999e-05 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.003432+0000", 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4726e-05 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004160+0000", 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072798800000000005 2024-09-17T07:47:39.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:55.004196+0000", 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5663000000000001e-05 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:39.875 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:39.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:39.939 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:39.938+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:39.978 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:40.218 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:40.218 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:40.218 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:40.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:40.220 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:40.219+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:40.320 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:40.320+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:40.323 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:40.323 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:40.323 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:40.323 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:40.323 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:40.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:40.421 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:40.421+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:40.474 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.472731572000001, 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.475 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:40.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:40.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:40.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.459958039, 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.482 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:40.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:40.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.456750927000002, 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:40.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:40.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.456280559, 2024-09-17T07:47:40.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:40.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.496 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.455697360000002, 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:40.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:40.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.455206838000002, 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.452690105000002, 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:40.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:40.510 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.441743522000003, 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:40.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.303964327000003, 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:40.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.519 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.303506573, 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.988503125000001, 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:40.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:40.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.985568897999997, 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.529 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.985115739999998, 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:40.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.537 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.984679706999998, 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.984102171000004, 2024-09-17T07:47:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.983557722, 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.983174411, 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.979900469, 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.570555173999999, 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:40.564 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.568589873999997, 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:40.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:40.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:40.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:40.569 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:40.570 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:40.571 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:40.570+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:40.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:47:40.671 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:40.671+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:40.676 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:40.676 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:40.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:40.772 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:40.772+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:40.828 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:40.828 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:40.828 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:40.829 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:40.829 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:40.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:40.879 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:40.878+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:40.974 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:40.974+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:40.979 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:40.979 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:40.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:40.979 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:40.979 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:40.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:41.075 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:41.074+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:41.097 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:41.097 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:41.097 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:41.097 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:41.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:41.175 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:41.175+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:41.204 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:41.204 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:41.204 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:41.204 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:41.204 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:41.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:41.277 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:41.277+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:41.311 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:41.311 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:41.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:41.311 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:41.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.311 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.150049842000001, 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:41.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.317 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149981488999998, 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:41.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149923383000001, 2024-09-17T07:47:41.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:41.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149902001000001, 2024-09-17T07:47:41.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:41.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149887085000003, 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:41.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149835435999996, 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:41.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.14977992, 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149724364000001, 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:41.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:41.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149651661999997, 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:41.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:41.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:41.364 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.149636788999999, 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:41.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:41.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821603285999998, 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:41.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:41.376 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821577122000001, 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:41.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:41.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:41.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:41.382 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821556282000003, 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:41.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:41.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821476038, 2024-09-17T07:47:41.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:41.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:41.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:41.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821463057000003, 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.394 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:41.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:41.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:41.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821446879, 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.400 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:41.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:41.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.821380388000001, 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.407735568, 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:41.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.407721926999997, 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:41.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:41.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.407684097999997, 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:41.424 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:41.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:41.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:41.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.429 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:41.430 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:41.430 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:41.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:41.432 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:41.431+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:41.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:41.547 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:41.547+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:41.570 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:41.570 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:41.570 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:41.570 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:41.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:41.594 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:47:41.629 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:47:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:47:41.632 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:41.632+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:41.645 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:41.645 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:41.660 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:41.661 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:41.676 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:41.676 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:41.676 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:41.677 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:41.677 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:41.677 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:41.677 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:41.685 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:41.692 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:41.692 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:41.708 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:41.733 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:41.733+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:41.782 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:41.782 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:41.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:41.875 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:41.875+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:41.930 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:41.930 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:41.935 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:41.935+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:41.938 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:42.036 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:42.036+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:42.076 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.076 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:42.137 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:42.137+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:42.181 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.182 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:47:42.238 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:42.238+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:42.300 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.300 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:47:42.339 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:42.339+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:42.410 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.410 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.418 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:47:42.462 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:42.462+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:42.519 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.519 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:42.569 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:42.569+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:42.642 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:42.642+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:42.671 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.671 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:42.743 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:42.743+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:42.780 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.780 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.789 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:42.844 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:42.843+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:42.891 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:42.891 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:42.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:42.944 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:42.944+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:43.041 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.042 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.045 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:43.044+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:43.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:43.146 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:43.146+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:43.192 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.192 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:43.247 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:43.247+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:43.303 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.303 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:43.348 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:43.347+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:43.415 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.415 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:43.449 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:47:43.448+0000 7fbfca5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31871) UID: 0 2024-09-17T07:47:43.521 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.521 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.529 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:43.550 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:43.549+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:43.597 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:43.597 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:47:43.597 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5] 2024-09-17T07:47:43.598 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T07:47:43.598 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:47:43.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:47:43.625 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T07:47:43.625 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-09-17T07:47:43.625 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:47:43.625 DEBUG:teuthology.orchestra.run.smithi026:> type ceph-objectstore-tool 2024-09-17T07:47:43.627 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.627 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.636 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:43.679 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:43.679+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:43.682 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T07:47:43.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-17T07:47:43.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:43.733 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.733 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:43.767 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:43.767+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:43.852 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:43.852+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:43.881 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:43.881 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:43.890 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:43.953 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:43.952+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:44.026 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.027 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:44.078 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:44.078+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:44.137 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.137 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:44.154 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:44.154+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:47:44.246+0000 7f9a3cae8200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-17T07:47:44.254 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:44.254+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:44.287 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.288 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:44.380 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:44.380+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:44.438 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.438 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:44.458 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:44.458+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:44.559 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:44.559+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:44.587 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.587 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:47:44.660 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:44.660+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:44.699 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.699 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:44.761 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:44.761+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:44.812 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.812 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:44.866 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:44.866+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:44.922 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:44.922 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:44.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:44.963 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:44.963+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:45.064 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:45.064+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:45.070 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.070 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:45.165 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:45.164+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:45.216 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.216 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:45.266 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:45.265+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:45.362 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.362 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.366 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:45.365+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:45.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:45.468 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:45.466+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:45.509 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.509 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:45.569 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:45.568+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:45.658 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.658 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.666 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:45.669 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:45.669+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:45.764 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.764 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-17T07:47:45.765+0000 7f9a3cae8200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-17T07:47:45.770 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:45.770+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:45.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:45.869 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.869 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.871 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:45.871+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:45.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:45.973 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:45.972+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:45.975 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:45.975 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:45.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:46.074 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:46.073+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.089 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:46.175 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:46.174+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:46.239 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.239 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:46.276 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:46.276+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:46.283 INFO:teuthology.orchestra.run.smithi026.stdout:3.3 2024-09-17T07:47:46.284 INFO:teuthology.orchestra.run.smithi026.stdout:3.1 2024-09-17T07:47:46.284 INFO:teuthology.orchestra.run.smithi026.stdout:1.6 2024-09-17T07:47:46.284 INFO:teuthology.orchestra.run.smithi026.stdout:1.5 2024-09-17T07:47:46.284 INFO:teuthology.orchestra.run.smithi026.stdout:1.2 2024-09-17T07:47:46.284 INFO:teuthology.orchestra.run.smithi026.stdout:1.1 2024-09-17T07:47:46.284 INFO:teuthology.orchestra.run.smithi026.stdout:1.0 2024-09-17T07:47:46.351 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.351 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:46.377 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:46.376+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:46.478 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:46.477+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:46.501 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.501 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:46.579 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:46.579+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:46.612 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.612 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.617 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:47:46.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:46.681 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:46.680+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:46.765 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.765 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:46.781 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:46.781+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:46.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 3.3 --file /var/log/ceph/exp.3.3.0 2024-09-17T07:47:46.876 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:46.877 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:46.883 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:46.882+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:46.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:46.983 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:46.982+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:47.023 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.023 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:47.116 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:47.115+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:47.169 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.169 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:47.186 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:47.185+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:47.286 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:47.285+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:47.316 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.317 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:47.387 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:47.387+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:47.464 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.464 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:47.517 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:47.516+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:47.570 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.570 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.578 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:47.589 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:47.589+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:47:47.690 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:47.688+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:47.716 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.717 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:47.791 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:47.790+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:47.862 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.863 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.871 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:47.914 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:47.913+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:47.938 INFO:teuthology.orchestra.run.smithi026.stderr:Exporting 3.3 info 3.3( empty local-lis/les=16/17 n=0 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-17T07:47:47.938 INFO:teuthology.orchestra.run.smithi026.stderr:Export successful 2024-09-17T07:47:47.967 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:47.967 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:47.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:48.017 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:48.016+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:48.081 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.081 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:48.092 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:48.092+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:48.190 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.191 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.193 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:48.192+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:48.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:48.294 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:48.293+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:48.301 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.301 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:48.395 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:48.394+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:48.452 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.453 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.459 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:48.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 3.3 2024-09-17T07:47:48.496 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:48.495+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:48.597 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:48.597+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:48.609 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.609 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:48.698 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:48.697+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:48.814 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:48.813+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:47:48.869 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:48.869 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:48.878 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:48.878 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:48.887 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:48.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:48.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:48.900 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:48.899+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:48.903 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:48.903 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:48.919 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:48.919 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:48.935 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:48.935 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:47:48.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:47:49.001 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:48.999+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:49.025 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.025 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:49.025 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:49.026 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:49.026 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:49.101 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:49.101+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:49.172 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.011245492, 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.011177138999997, 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:49.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:49.184 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.011119033, 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:49.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.011097651, 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:49.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.011082735000002, 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.196 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:49.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:49.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.201 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.011031086000003, 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.010975569999999, 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:49.207 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:49.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.010920014, 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.213 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:49.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:49.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.010847312000003, 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:49.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:49.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.010832438999998, 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:49.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:49.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682798935999998, 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:49.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:49.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.241 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682772772, 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:49.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:49.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:49.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:49.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.248 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.248 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:49.248 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682751932000002, 2024-09-17T07:47:49.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:49.248 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.249 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.249 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:49.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:49.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682671687999999, 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:49.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:49.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.262 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682658707000002, 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:49.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:49.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:49.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682642528999999, 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:49.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:49.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:49.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:49.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.682576038000001, 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.278 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:49.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:49.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:49.285 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.268931217999999, 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:49.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:49.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:49.291 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.268917577000003, 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:49.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:49.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:49.298 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.268879748000003, 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:49.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.305 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.306 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.306 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.306 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:49.309 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:49.307+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.408 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:49.407+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:49.415 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:49.509 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:49.509+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:49.553 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.554 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:49.554 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:49.554 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:49.554 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.562 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: marking collection for removal 2024-09-17T07:47:49.598 INFO:teuthology.orchestra.run.smithi026.stdout:setting '_remove' omap key 2024-09-17T07:47:49.598 INFO:teuthology.orchestra.run.smithi026.stdout:finish_remove_pgs 3.3_head removing 3.3 2024-09-17T07:47:49.605 INFO:teuthology.orchestra.run.smithi026.stdout:Remove successful 2024-09-17T07:47:49.610 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:49.609+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:49.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:47:49.661 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.661 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.655802719999997, 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:49.662 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:49.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:49.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.655330372999998, 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:49.666 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:49.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:49.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.66565739, 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:49.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:49.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:49.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:49.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.663210200999998, 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:49.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:49.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:49.680 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.659557935999999, 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:49.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:49.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:49.687 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.656648394999998, 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:49.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:49.691 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:49.733 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:49.732+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:49.812 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:49.810+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:49.830 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.830 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:49.830 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:49.830 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:49.922 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:49.921+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:49.975 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:49.976 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:49.976 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:49.976 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:49.976 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:49.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:50.025 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:50.024+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:50.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.3.0 2024-09-17T07:47:50.115 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:50.114+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:50.122 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.121436185999997, 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:50.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:50.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:50.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:50.127 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:50.130 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:50.215 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:50.215+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:50.258 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:50.259 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:50.259 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:50.259 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:50.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:50.351 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:50.351+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:50.409 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:50.409 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:50.409 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:50.409 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:50.409 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:50.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:50.418 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:50.417+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:50.518 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:50.518+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:50.566 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.564626637000003, 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:50.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:50.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:50.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:50.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:50.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:50.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.551853104000003, 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:50.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:50.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.548645991999997, 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:50.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:50.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:50.583 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.548175624000002, 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:50.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:50.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.547592424999998, 2024-09-17T07:47:50.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.547101902999998, 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.544585169999998, 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.533638586999999, 2024-09-17T07:47:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.395859391999998, 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.395401638000003, 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.080398189999997, 2024-09-17T07:47:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.077463963, 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.077010805, 2024-09-17T07:47:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.076574772000001, 2024-09-17T07:47:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.075997235999999, 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.075452787000003, 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.075069476000003, 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.071795534000003, 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.662450239000002, 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.660484939, 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:50.662 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:50.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:50.665 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:47:50.665 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:50.663+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:50.764 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:50.764+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:50.807 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:50.808 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:50.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:50.865 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:50.864+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:50.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:50.957 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:50.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:50.957 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:50.958 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:50.958 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:50.966 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:50.965+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:50.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:47:51.059 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:51.059 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:51.067 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:51.066+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:51.075 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:51.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:51.091 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:51.107 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:51.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:51.111 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.111 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:51.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:47:51.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:47:51.168 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:51.167+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:51.229 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.229 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:51.257 INFO:teuthology.orchestra.run.smithi026.stdout:get_pg_num_history pg_num_history pg_num_history(e38 pg_nums {1={12=8},2={12=1},3={16=4}} deleted_pools ) 2024-09-17T07:47:51.257 INFO:teuthology.orchestra.run.smithi026.stdout:Importing pgid 3.3 2024-09-17T07:47:51.257 INFO:teuthology.orchestra.run.smithi026.stdout:write_pg epoch 17 info 3.3( empty local-lis/les=16/17 n=0 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-17T07:47:51.258 INFO:teuthology.orchestra.run.smithi026.stdout:Import successful 2024-09-17T07:47:51.280 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:51.278+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:51.369 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:51.369+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:51.377 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.377 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:51.470 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:51.469+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:51.484 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.484 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:51.576 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:51.575+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:51.682 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:51.681+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:51.774 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:51.773+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:51.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /var/log/ceph/exp.3.3.0 2024-09-17T07:47:51.779 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.779 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:51.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:47:51.875 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:51.874+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:51.884 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:51.885 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:51.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:51.977 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:51.976+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:52.031 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.032 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:52.077 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:52.077+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:47:52.175 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":42,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:47:50.605126+0000","last_up_change":"2024-09-17T07:47:44.594264+0000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":29,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":31884},{"type":"v1","addr":"172.21.15.26:6801","nonce":31884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":31884},{"type":"v1","addr":"172.21.15.26:6803","nonce":31884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":31884},{"type":"v1","addr":"172.21.15.26:6807","nonce":31884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":31884},{"type":"v1","addr":"172.21.15.26:6805","nonce":31884}]},"public_addr":"172.21.15.26:6801/31884","cluster_addr":"172.21.15.26:6803/31884","heartbeat_back_addr":"172.21.15.26:6807/31884","heartbeat_front_addr":"172.21.15.26:6805/31884","state":["exists"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":0.3944549560546875,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":31604},{"type":"v1","addr":"172.21.15.94:6801","nonce":31604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":31604},{"type":"v1","addr":"172.21.15.94:6803","nonce":31604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":31604},{"type":"v1","addr":"172.21.15.94:6807","nonce":31604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":31604},{"type":"v1","addr":"172.21.15.94:6805","nonce":31604}]},"public_addr":"172.21.15.94:6801/31604","cluster_addr":"172.21.15.94:6803/31604","heartbeat_back_addr":"172.21.15.94:6807/31604","heartbeat_front_addr":"172.21.15.94:6805/31604","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":31606},{"type":"v1","addr":"172.21.15.94:6817","nonce":31606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":31606},{"type":"v1","addr":"172.21.15.94:6819","nonce":31606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":31606},{"type":"v1","addr":"172.21.15.94:6823","nonce":31606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":31606},{"type":"v1","addr":"172.21.15.94:6821","nonce":31606}]},"public_addr":"172.21.15.94:6817/31606","cluster_addr":"172.21.15.94:6819/31606","heartbeat_back_addr":"172.21.15.94:6823/31606","heartbeat_front_addr":"172.21.15.94:6821/31606","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.5","osds":[1,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:47:52.177 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.178 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.178 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:52.177+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:52.186 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-17T07:47:52.186 DEBUG:teuthology.orchestra.run.smithi026:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op apply-layout-settings --pool rbd 2024-09-17T07:47:52.188 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:52.278 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:52.277+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:52.284 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.285 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:52.379 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:52.379+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:52.395 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.395 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.404 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:52.480 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:52.480+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.505 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.514 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:52.581 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:52.581+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:52.621 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.621 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:52.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:47:52.682 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:52.681+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:52.776 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.776 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.782 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:52.781+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:52.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:52.884 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:52.883+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:52.887 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.887 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:52.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:52.985 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:52.984+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:52.997 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:52.998 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:53.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:53.086 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:53.084+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:53.105 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:53.105 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:53.114 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:53.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:53.186 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:53.186+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:47:53.246 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:53.252 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:53.252 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:53.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:53.262 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:53.262 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:53.277 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:53.278 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:53.287 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:53.286+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:53.293 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:53.308 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:53.308 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:47:53.311 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:47:53.343 INFO:teuthology.orchestra.run.smithi026.stderr:Nothing to do for non-filestore backend 2024-09-17T07:47:53.360 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199551593999999, 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.361 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:53.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:53.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.367 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199483241000003, 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:53.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199425134999998, 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.374 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:53.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:53.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199403752999999, 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:53.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:53.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199388837000001, 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:53.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:53.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199337188000001, 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:53.392 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:53.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:53.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199281671999998, 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.399 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:53.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:53.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:53.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199226115999998, 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:53.405 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:53.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:53.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199153414000001, 2024-09-17T07:47:53.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:53.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:53.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.199138541000003, 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:53.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.871105038000003, 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:53.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:53.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.871078873999998, 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.429 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:53.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:53.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.871058034000001, 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:53.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:53.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.870977789999998, 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:53.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:53.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:53.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.448 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.870964809, 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:53.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.870948630999997, 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:53.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.460 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.870882139999999, 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.457237319999997, 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:53.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.457223679000002, 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:53.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:53.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:53.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:53.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.457185850000002, 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:53.478 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.484 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:53.486 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:53.486 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:53.485+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:53.587 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:53.587+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:53.618 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:47:53.626 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:53.626 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:53.634 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:53.687 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:53.686+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:53.732 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:53.733 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:53.733 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:53.733 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:53.733 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:53.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:53.788 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:53.788+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:53.862 DEBUG:teuthology.orchestra.run.smithi026:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op apply-layout-settings --pool .mgr 2024-09-17T07:47:53.879 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.874262139999999, 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.873789793, 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.884116810000002, 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.881669621, 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.878017356000001, 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.875107815, 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:53.913 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:53.915 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:53.915 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:53.914+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:54.014 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:54.014 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:54.015 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:54.014+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:54.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:54.117 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:54.116+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:54.159 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:54.160 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:54.160 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:54.160 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:54.160 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:54.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:54.218 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:54.218+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:54.264 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.263577939000001, 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:54.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:54.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:54.269 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:54.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:54.319 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:54.318+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:54.386 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:54.387 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:54.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:54.420 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:54.419+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:54.496 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:54.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:54.496 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:54.496 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:54.497 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:54.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:54.548 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:54.548+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.605888915000001, 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:54.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:54.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.593115382000001, 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:54.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:54.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.589908270000002, 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:54.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.589437902, 2024-09-17T07:47:54.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:54.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.588854703000003, 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:54.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:54.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.588364181000003, 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:54.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.585847448000003, 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.637 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.574900865000004, 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:54.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:54.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:54.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.437121670000003, 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:54.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:54.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.436663916000001, 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:54.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:54.655 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.121660468000002, 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:54.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:54.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.118726240999997, 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:54.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:54.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.118273082999998, 2024-09-17T07:47:54.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:54.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.117837049999999, 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:54.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.117259513999997, 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:54.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.116715065000001, 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:54.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.116331754000001, 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:54.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:54.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.687 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.113057812000001, 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:54.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.703712517, 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.701747216999998, 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:54.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:54.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:54.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:54.701 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:54.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:54.704 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:54.702+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:54.803 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:54.803+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:54.845 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:54.845 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:54.846 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:54.846 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:54.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:54.905 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:54.903+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:54.995 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:54.996 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:54.996 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:54.996 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:54.996 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:55.002 INFO:teuthology.orchestra.run.smithi026.stderr:Nothing to do for non-filestore backend 2024-09-17T07:47:55.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:55.006 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:55.006+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:55.107 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:55.108 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:55.108 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:55.107+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:55.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:55.208 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:55.208+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:55.222 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:55.222 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:55.222 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:55.222 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:55.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:55.310 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:55.309+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:55.312 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:55.329 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:55.329 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:55.329 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:55.330 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:55.330 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:55.338 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:55.410 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:55.409+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:47:55.429 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:55.439 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:55.439 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:55.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278384287000002, 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:55.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:55.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278315933999998, 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:55.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.451 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278257828000001, 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:55.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:55.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278236446000001, 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.457 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278221530000003, 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.463 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:55.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:55.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278169880999997, 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:55.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:55.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:55.474 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278114365, 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:55.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:55.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.278058809000001, 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:55.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:55.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.277986106999997, 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:55.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:55.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.277971233999999, 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.492 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:55.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949937730999999, 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:55.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:55.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949911567000001, 2024-09-17T07:47:55.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:55.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949890727000003, 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:55.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949810483, 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.515 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:55.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949797502000003, 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:55.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:55.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949781324, 2024-09-17T07:47:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:55.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.949714833000002, 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:55.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:55.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.536070013, 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:55.538 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.536056371999997, 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:55.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:55.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.536018542999997, 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:55.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:55.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:55.555 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:55.556 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:55.557 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:55.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:55.558 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:55.556+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:55.559 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:55.559 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:55.559 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:47:55.560 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:55.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:47:55.561 DEBUG:teuthology.orchestra.run.smithi026:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op apply-layout-settings --pool cache 2024-09-17T07:47:55.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:47:55.657 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:55.656+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:55.697 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:55.698 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:55.706 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:55.791 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:55.790+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:55.843 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:55.843 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:55.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:55.892 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:55.891+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:55.960 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:55.959+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:55.990 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:55.990 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:55.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:56.061 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:56.061+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:56.135 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.136 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:56.161 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:56.161+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:56.242 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.243 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.251 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:56.294 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:56.293+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:56.348 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.348 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:56.400 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:56.400+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:56.464 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:56.464+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:56.466 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.466 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:56.566 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:56.565+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:56.617 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.617 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:56.628 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:47:56.655 INFO:teuthology.orchestra.run.smithi026.stderr:Nothing to do for non-filestore backend 2024-09-17T07:47:56.666 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:56.665+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:56.756 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.756 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:56.766 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:56.766+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:56.867 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.867 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.867 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:56.866+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:56.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:56.968 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:56.967+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:56.976 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:56.976 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:56.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:57.068 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:57.068+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:57.087 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:57.087 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:57.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:57.169 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:57.168+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:57.205 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:57.205 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:57.215 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:57.299 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:57.298+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:57.354 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:57.354 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:57.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:57.404 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:57.403+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:57.471 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:57.471+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:57.502 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:57.502 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:57.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:57.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:47:57.572 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:57.571+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:57.610 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:47:57.610 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:47:57.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:57.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:47:57.673 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:57.673+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:57.685 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:57.701 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:57.717 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:57.734 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:57.749 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:57.749 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:47:57.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:47:57.759 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:57.759 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:57.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:57.811 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:57.810+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:57.866 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.861311293, 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186868, 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:57.867 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005097+0000", 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005099+0000", 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.813e-06 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005105+0000", 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3979999999999998e-06 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005114+0000", 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.55e-06 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005123+0000", 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5050000000000007e-06 2024-09-17T07:47:57.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005175+0000", 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2485999999999999e-05 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005284+0000", 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000109116 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.860838946000001, 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016912000000000001, 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 3 2024-09-17T07:47:57.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005570+0000", 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005571+0000", 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005576+0000", 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2950000000000001e-06 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005587+0000", 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1399e-05 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005593+0000", 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005635+0000", 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2209000000000002e-05 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.005739+0000", 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103902 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.871165963000003, 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025488780000000001, 2024-09-17T07:47:57.876 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 5 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995243+0000", 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995251+0000", 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6170000000000007e-06 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995262+0000", 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0944999999999999e-05 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995278+0000", 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6154000000000002e-05 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995285+0000", 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4150000000000004e-06 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995347+0000", 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2550000000000003e-05 2024-09-17T07:47:57.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.995354+0000", 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0299999999999996e-06 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996635+0000", 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012805589999999999 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996736+0000", 2024-09-17T07:47:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100924 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.996831+0000", 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4650999999999999e-05 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997436+0000", 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060566500000000002 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997744+0000", 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000307304 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997771+0000", 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7124e-05 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997792+0000", 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0939999999999999e-05 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:5 1.2 e14/12 ondisk, result = 0)", 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.868718774000001, 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112502, 2024-09-17T07:47:57.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997690+0000", 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997691+0000", 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997694+0000", 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.157e-06 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997703+0000", 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1959999999999997e-06 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997714+0000", 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997736+0000", 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2395000000000002e-05 2024-09-17T07:47:57.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997741+0000", 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2080000000000002e-06 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.997802+0000", 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1853000000000006e-05 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4187.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.865066509000002, 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003072633, 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4187", 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 7 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:57.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001342+0000", 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001344+0000", 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.736e-06 2024-09-17T07:47:57.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001351+0000", 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5870000000000004e-06 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001367+0000", 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5631999999999999e-05 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001374+0000", 2024-09-17T07:47:57.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0679999999999999e-06 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001426+0000", 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1952e-05 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op must wait for map", 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.001431+0000", 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7269999999999998e-06 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002932+0000", 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001500899 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.002992+0000", 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0371999999999997e-05 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 0", 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003085+0000", 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2758999999999997e-05 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.003774+0000", 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068909899999999998 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004321+0000", 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00054724999999999999 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004372+0000", 2024-09-17T07:47:57.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0319999999999999e-05 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004415+0000", 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3232000000000002e-05 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.894 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4187.0:7 1.2 e15/12 ondisk, result = 0)", 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.862156968000001, 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000179428, 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:57.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004252+0000", 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004253+0000", 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004258+0000", 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8409999999999996e-06 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004268+0000", 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6229999999999992e-06 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004273+0000", 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0860000000000001e-06 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004306+0000", 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2468999999999997e-05 2024-09-17T07:47:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004314+0000", 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3180000000000007e-06 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:55.004431+0000", 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117261 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:57.898 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:57.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:47:57.900 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:57.898+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:58.000 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:58.000+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:58.038 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:58.038 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:58.039 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:58.039 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:58.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:47:58.101 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:58.101+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:58.144 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:58.144 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:58.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:47:58.203 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:58.202+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:58.249 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:58.249 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:58.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:58.249 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.248504523999998, 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:47:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:47:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:47:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:58.254 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:58.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:47:58.303 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:58.303+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:58.363 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:58.364 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:58.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:47:58.404 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:58.404+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:58.505 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:58.505+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:58.514 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:58.514 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:58.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:47:58.607 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:58.606+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:58.627 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.625407573000004, 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:58.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:47:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:17 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'7, mlcod=13'5)", 2024-09-17T07:47:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.612634040000003, 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088150499999999998, 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013357+0000", 2024-09-17T07:47:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013358+0000", 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8800000000000002e-07 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013376+0000", 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8335999999999999e-05 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013389+0000", 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3164e-05 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013394+0000", 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9880000000000004e-06 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013450+0000", 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400999999999998e-05 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.013467+0000", 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6064000000000001e-05 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014210+0000", 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074369599999999996 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.014238+0000", 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8068e-05 2024-09-17T07:47:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.609426928000005, 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093472499999999997, 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016564+0000", 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016565+0000", 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016696+0000", 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131552 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016710+0000", 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3308e-05 2024-09-17T07:47:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016716+0000", 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016768+0000", 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2441000000000002e-05 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.016785+0000", 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6884e-05 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017452+0000", 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066729799999999998 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017499+0000", 2024-09-17T07:47:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.608956559999996, 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:47:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:22 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'12, mlcod=13'8)", 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.608373361000005, 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000935767, 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017617+0000", 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017621+0000", 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8060000000000001e-06 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017715+0000", 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3912000000000005e-05 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017730+0000", 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-05 2024-09-17T07:47:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017736+0000", 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017782+0000", 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5380999999999998e-05 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017804+0000", 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.177e-05 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018523+0000", 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071952400000000003 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018553+0000", 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0086e-05 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:23 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'13, mlcod=13'8)", 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.607882838999998, 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083439699999999996, 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018108+0000", 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018117+0000", 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5830000000000007e-06 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018184+0000", 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7818999999999996e-05 2024-09-17T07:47:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018197+0000", 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2349e-05 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018201+0000", 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2220000000000004e-06 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018227+0000", 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6219000000000001e-05 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018240+0000", 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3329000000000001e-05 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018913+0000", 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067275699999999996 2024-09-17T07:47:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018942+0000", 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9119e-05 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:26 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.605366106000005, 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078866099999999996, 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020625+0000", 2024-09-17T07:47:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020626+0000", 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020643+0000", 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7447000000000001e-05 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020656+0000", 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2696e-05 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020660+0000", 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7010000000000004e-06 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020707+0000", 2024-09-17T07:47:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6628999999999997e-05 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.020720+0000", 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3192e-05 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021382+0000", 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066183400000000003 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.021413+0000", 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1365000000000002e-05 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:33 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'19, mlcod=13'17)", 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.594419522999999, 2024-09-17T07:47:58.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000811669, 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031571+0000", 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031573+0000", 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.446e-06 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031610+0000", 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7493000000000002e-05 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031635+0000", 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4493000000000001e-05 2024-09-17T07:47:58.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031645+0000", 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0137e-05 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031765+0000", 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120418 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.031844+0000", 2024-09-17T07:47:58.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8405999999999994e-05 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032351+0000", 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050726099999999995 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.032383+0000", 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2014999999999998e-05 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:53 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.456640328000006, 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082065399999999998, 2024-09-17T07:47:58.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169350+0000", 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169351+0000", 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169396+0000", 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4558000000000003e-05 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169407+0000", 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0890000000000001e-05 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169410+0000", 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8510000000000001e-06 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169449+0000", 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8562000000000001e-05 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169462+0000", 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2672e-05 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170144+0000", 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000682717 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170171+0000", 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6721999999999999e-05 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:54 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-17T07:47:58.671 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.456182573999996, 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086793899999999997, 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169808+0000", 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169809+0000", 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169888+0000", 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9562999999999997e-05 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169900+0000", 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1080000000000001e-05 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169903+0000", 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5420000000000001e-06 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169927+0000", 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3788e-05 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.169938+0000", 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1005999999999999e-05 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170650+0000", 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071242899999999999 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.170676+0000", 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5894999999999999e-05 2024-09-17T07:47:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:59 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'41, mlcod=13'39)", 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.141179125999997, 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095757199999999996, 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484812+0000", 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484814+0000", 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4669999999999998e-06 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484851+0000", 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6983000000000001e-05 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484883+0000", 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1689000000000001e-05 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.484907+0000", 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4105999999999999e-05 2024-09-17T07:47:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485008+0000", 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010145399999999999 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485028+0000", 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0034000000000002e-05 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485727+0000", 2024-09-17T07:47:58.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069837000000000002 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.485769+0000", 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:64 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'40)", 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.138244899, 2024-09-17T07:47:58.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081277900000000002, 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487746+0000", 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487747+0000", 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4499999999999997e-07 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487754+0000", 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2729999999999997e-06 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487764+0000", 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0614999999999999e-05 2024-09-17T07:47:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487769+0000", 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2509999999999996e-06 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487794+0000", 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5568999999999999e-05 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.487806+0000", 2024-09-17T07:47:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1681999999999999e-05 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488525+0000", 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071938099999999995 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488559+0000", 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3362999999999999e-05 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:65 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'40)", 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.137791741000001, 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097704999999999992, 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488199+0000", 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488200+0000", 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7999999999999995e-07 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488208+0000", 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1089999999999998e-06 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488218+0000", 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9350000000000006e-06 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488229+0000", 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0779e-05 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488263+0000", 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4326000000000003e-05 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488284+0000", 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0859000000000001e-05 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489148+0000", 2024-09-17T07:47:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086391500000000004 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489176+0000", 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8447000000000001e-05 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:66 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'44, mlcod=13'40)", 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.137355708000001, 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000908533, 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488635+0000", 2024-09-17T07:47:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488636+0000", 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488696+0000", 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488704+0000", 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3219999999999993e-06 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488707+0000", 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2600000000000001e-06 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488740+0000", 2024-09-17T07:47:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2910000000000002e-05 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.488759+0000", 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8462999999999999e-05 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489521+0000", 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076230099999999995 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489544+0000", 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2623999999999999e-05 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:67 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'45, mlcod=13'40)", 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.136778172000007, 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081657500000000005, 2024-09-17T07:47:58.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489213+0000", 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489217+0000", 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.233e-06 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489226+0000", 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000008e-06 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489230+0000", 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8920000000000002e-06 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489261+0000", 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1155000000000002e-05 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489278+0000", 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490002+0000", 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072449000000000005 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490029+0000", 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7169e-05 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'46, mlcod=13'42)", 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.136233723000004, 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085726499999999996, 2024-09-17T07:47:58.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489757+0000", 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489758+0000", 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489768+0000", 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0753e-05 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489776+0000", 2024-09-17T07:47:58.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9170000000000006e-06 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489780+0000", 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5779999999999999e-06 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489803+0000", 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3459e-05 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.489817+0000", 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3645999999999999e-05 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490592+0000", 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077543399999999998 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490614+0000", 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1985e-05 2024-09-17T07:47:58.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'47, mlcod=13'42)", 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.135850411999996, 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095978100000000005, 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490140+0000", 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490141+0000", 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7399999999999999e-07 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490201+0000", 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.995e-05 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490209+0000", 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490214+0000", 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6410000000000001e-06 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490243+0000", 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9414e-05 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.490255+0000", 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2221e-05 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491066+0000", 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081108000000000005 2024-09-17T07:47:58.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.491100+0000", 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3827e-05 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:75 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'49)", 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.132576470000004, 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084009999999999998, 2024-09-17T07:47:58.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493414+0000", 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493415+0000", 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493484+0000", 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8830000000000003e-05 2024-09-17T07:47:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493491+0000", 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5800000000000003e-06 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493504+0000", 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2298e-05 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493526+0000", 2024-09-17T07:47:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399999999999999e-05 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.493540+0000", 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4089e-05 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494220+0000", 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067947000000000005 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.494254+0000", 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4904000000000003e-05 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:90 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'64, mlcod=13'60)", 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.723231174999995, 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000902577, 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902760+0000", 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902842+0000", 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2114999999999996e-05 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902850+0000", 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9300000000000003e-06 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902856+0000", 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0220000000000001e-06 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902876+0000", 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0027999999999999e-05 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.902886+0000", 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903632+0000", 2024-09-17T07:47:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074619900000000002 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.903662+0000", 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9742000000000001e-05 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:93 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'63)", 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.721265875, 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092947799999999997, 2024-09-17T07:47:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904725+0000", 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3700000000000001e-07 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904771+0000", 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5939000000000002e-05 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904778+0000", 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1629999999999999e-06 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904782+0000", 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0309999999999999e-06 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904802+0000", 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.904814+0000", 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1229e-05 2024-09-17T07:47:58.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905614+0000", 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080060600000000004 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.905654+0000", 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0243000000000001e-05 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:47:58.721 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:58.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:47:58.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:47:58.724 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:58.723+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:58.824 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:58.823+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:58.868 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:47:58.868 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:58.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:47:58.963 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:58.963+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:59.022 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:59.022 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:47:59.022 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:47:59.022 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:47:59.022 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:59.025 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:47:59.024+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:47:59.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:47:59.125 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:59.125+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:59.174 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:47:59.174 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:47:59.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:47:59.175 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-09-17T07:47:59.175 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:47:59.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:47:59.226 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:47:59.226+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:47:59.293 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:59.293 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:59.293 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:59.293 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:59.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:47:59.345 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:59.344+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:59.400 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:59.400 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:59.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:47:59.449 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:59.449+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:59.529 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:47:59.528+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387680301000003, 2024-09-17T07:47:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:47:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:47:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387611948, 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:47:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:47:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387553842000003, 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:47:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:47:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387532460000003, 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:47:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:47:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387517544000005, 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:47:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387465895000005, 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:47:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:47:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387410379000002, 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.587 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:59.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:47:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387354822999995, 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:47:59.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:47:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387282120999998, 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:59.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:47:59.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.387267248000001, 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:47:59.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.610 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059233745, 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.611 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:47:59.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:47:59.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059207580999995, 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:47:59.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.622 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059186741000005, 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:59.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:47:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.628 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059106497000002, 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:47:59.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:47:59.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059093516000004, 2024-09-17T07:47:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:47:59.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:47:59.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059077337999994, 2024-09-17T07:47:59.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:47:59.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:47:59.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:47:59.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.059010846999996, 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:47:59.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:47:59.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:47:59.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.652 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:92 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.645366026999994, 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003971891, 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 92 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901736+0000", 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901738+0000", 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0219999999999999e-06 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901742+0000", 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6449999999999998e-06 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901747+0000", 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0679999999999996e-06 2024-09-17T07:47:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.903979+0000", 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022317209999999999 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904048+0000", 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8842000000000003e-05 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904163+0000", 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115151 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905146+0000", 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098372200000000003 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905631+0000", 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00048425999999999997 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905670+0000", 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9768000000000003e-05 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905708+0000", 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7388999999999997e-05 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.658 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:93 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.645352385999999, 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040502510000000004, 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901750+0000", 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.28e-07 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901775+0000", 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5403000000000001e-05 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901780+0000", 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1499999999999998e-06 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901783+0000", 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4990000000000001e-06 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904452+0000", 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026693979999999999 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904525+0000", 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2943999999999998e-05 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.904651+0000", 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012531400000000001 2024-09-17T07:47:59.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905716+0000", 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010650880000000001 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905756+0000", 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0259999999999997e-05 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905781+0000", 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5499000000000001e-05 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905800+0000", 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8468e-05 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.645314557000006, 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:47:59.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:47:59.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:47:59.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.671 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:47:59.671 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:47:59.671 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:59.673 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:47:59.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:47:59.673 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:59.671+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:47:59.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:47:59.798 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:59.797+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:59.812 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:59.812 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:59.812 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:47:59.812 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:59.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:47:59.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:47:59.873 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:47:59.873+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:47:59.878 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:47:59.879 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:47:59.894 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:47:59.894 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:47:59.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:47:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:47:59.918 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:47:59.918 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:47:59.918 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:47:59.918 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:47:59.918 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:47:59.925 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:47:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:47:59.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:47:59.941 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:47:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:47:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:47:59.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:47:59.974 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:47:59.973+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:00.024 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.024 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:00.076 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:00.075+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:00.171 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.171 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.176 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:00.175+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:00.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:00.278 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:00.277+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:00.317 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.318 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:00.379 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:00.379+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:00.464 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.464 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:00.480 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:00.479+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.541 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-17T07:48:00.542 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-17T07:48:00.546 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T07:48:00.579 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.579 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.580 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:00.580+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:00.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:00.656 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T07:48:00.658 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T07:48:00.659 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T07:48:00.681 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:00.680+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:00.730 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.730 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:00.783 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:00.781+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:00.881 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:00.881 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:00.882 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:00.882+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:00.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:00.983 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:00.982+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:01.031 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.031 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:01.084 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:01.083+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:01.141 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.141 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.149 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:01.184 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:01.184+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:01.251 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.251 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:48:01.306 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:01.306+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:01.357 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T07:48:01.368 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.368 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:48:01.418 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:01.417+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:01.425 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T07:48:01.466 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T07:48:01.468 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T07:48:01.468 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T07:48:01.486 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:01.485+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:01.506 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T07:48:01.510 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-17T07:48:01.510 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T07:48:01.516 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.516 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:48:01.527 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-09-17T07:48:01.528 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-09-17T07:48:01.528 DEBUG:teuthology.orchestra.run.smithi098:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-17T07:48:01.587 INFO:tasks.workunit.client.0.smithi098.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ TIMEOUT=300 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ MAX_TIMEOUT=15 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ PG_NUM=4 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ TMPDIR=/tmp 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-17T07:48:01.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-17T07:48:01.589 INFO:tasks.workunit.client.0.smithi098.stderr:++ type xmlstarlet 2024-09-17T07:48:01.589 INFO:tasks.workunit.client.0.smithi098.stderr:++ XMLSTARLET=xmlstarlet 2024-09-17T07:48:01.589 INFO:tasks.workunit.client.0.smithi098.stderr:+++ uname 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' Linux = FreeBSD ']' 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:++ SED=sed 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:++ AWK=awk 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:+++ stty -a 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:+++ head -1 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-17T07:48:01.590 INFO:tasks.workunit.client.0.smithi098.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T07:48:01.591 INFO:tasks.workunit.client.0.smithi098.stderr:++ termwidth= 2024-09-17T07:48:01.591 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-17T07:48:01.591 INFO:tasks.workunit.client.0.smithi098.stderr:++ DIFFCOLOPTS='-y ' 2024-09-17T07:48:01.591 INFO:tasks.workunit.client.0.smithi098.stderr:++ KERNCORE=kernel.core_pattern 2024-09-17T07:48:01.591 INFO:tasks.workunit.client.0.smithi098.stderr:++ EXTRA_OPTS= 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:++ test '' = TESTS 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ POOL=rbd 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ ANOTHER_POOL=new_default_pool27356 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ NS=ns 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ IMAGE=testrbdnbd27356 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ SIZE=64 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ DATA= 2024-09-17T07:48:01.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ setup 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns x 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -e CMakeCache.txt ']' 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo echo test sudo 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:01.595 INFO:tasks.workunit.client.0.smithi098.stderr:++ which echo 2024-09-17T07:48:01.604 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/echo 2024-09-17T07:48:01.604 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:01.604 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-09-17T07:48:01.608 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:01.608+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:01.613 INFO:tasks.workunit.client.0.smithi098.stdout:test sudo 2024-09-17T07:48:01.615 INFO:tasks.workunit.client.0.smithi098.stderr:+ trap cleanup INT TERM EXIT 2024-09-17T07:48:01.615 INFO:tasks.workunit.client.0.smithi098.stderr:++ mktemp -d 2024-09-17T07:48:01.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ TEMPDIR=/tmp/tmp.TyfTMF4aW5 2024-09-17T07:48:01.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ DATA=/tmp/tmp.TyfTMF4aW5/data 2024-09-17T07:48:01.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/urandom of=/tmp/tmp.TyfTMF4aW5/data bs=1M count=64 2024-09-17T07:48:01.662 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.662 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.670 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:01.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:48:01.711 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:01.711+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:01.788 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:01.787+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:01.809 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.810 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:01.819 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:01.819 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:01.819 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.179825 s, 373 MB/s 2024-09-17T07:48:01.819 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd namespace create rbd/ns 2024-09-17T07:48:01.883 INFO:tasks.workunit.client.0.smithi098.stderr:+ for ns in '' ${NS} 2024-09-17T07:48:01.883 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.TyfTMF4aW5/data testrbdnbd27356 2024-09-17T07:48:01.902 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:01.901+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:01.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:01.964 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:01.964 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:01.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:01.990 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:01.990+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:02.080 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:02.091 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:02.089+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:02.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:02.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:02.098 INFO:tasks.workunit.client.0.smithi098.stderr:+ for ns in '' ${NS} 2024-09-17T07:48:02.098 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.TyfTMF4aW5/data testrbdnbd27356 2024-09-17T07:48:02.112 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:02.128 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:48:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:48:02.140 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:02.140 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:02.144 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:02.145 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:02.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:02.162 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:02.162 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:02.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:02.176 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5] 2024-09-17T07:48:02.177 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:02.177 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.456400699929427 2024-09-17T07:48:02.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 0.456400699929427 2024-09-17T07:48:02.208 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:02.208+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:02.273 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph osd pool create new_default_pool27356 8 2024-09-17T07:48:02.292 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:02.291+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:02.306 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:02.306 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:02.306 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:02.306 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:02.314 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:02.393 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:02.393+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:02.453 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:02.453 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:02.454 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:02.454 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:02.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:02.494 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:02.494+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:02.559 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:02.559 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:02.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:02.559 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.557709557999999, 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:48:02.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:48:02.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:02.564 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.37712729699999997, 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:02.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:02.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.36898725900000001, 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.569 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:02.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.34936456700000001, 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.342039335, 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:02.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.582 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.33542723400000002, 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:02.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:02.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:02.587 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.331245911, 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:02.591 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.32893326699999997, 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:02.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:02.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:02.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.30388616299999999, 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:02.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:02.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.602 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.303548706, 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:02.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4801.0:60 1.7 e45/12 ondisk, result = 0)", 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.29303649599999998, 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012042, 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5099999999999999e-07 2024-09-17T07:48:02.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265799+0000", 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4870000000000004e-06 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265809+0000", 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0304e-05 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265814+0000", 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3959999999999999e-06 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265845+0000", 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1043000000000001e-05 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265850+0000", 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7150000000000003e-06 2024-09-17T07:48:02.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265914+0000", 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3724000000000002e-05 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:02.611 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:02.613 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:02.612+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:02.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:48:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 0.456401 (74d6) 2024-09-17T07:48:02.713 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:02.713+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:02.719 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:02.719 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:02.719 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:02.719 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:02.727 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:02.730 INFO:tasks.workunit.client.0.smithi098.stderr:pool 'new_default_pool27356' already exists 2024-09-17T07:48:02.740 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd pool init new_default_pool27356 2024-09-17T07:48:02.814 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:02.813+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:02.869 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:02.869 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:02.878 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:02.914 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:02.914+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:03.015 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:03.015+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:03.022 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:48:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 70.019850593000001, 2024-09-17T07:48:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:48:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:48:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:48:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:48:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:48:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:48:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:48:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:48:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:48:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:48:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:48:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:48:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:48:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:48:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:48:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:48:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:48:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:48:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:48:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:48:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:48:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:48:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 70.003399580000007, 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:48:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:48:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:6 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_assert in=19b,call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.88576310400000002, 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001221846, 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 6 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134674+0000", 2024-09-17T07:48:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0989999999999999e-06 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134681+0000", 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1179999999999997e-06 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134683+0000", 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8449999999999999e-06 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134706+0000", 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2849000000000001e-05 2024-09-17T07:48:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134755+0000", 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8547000000000001e-05 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134915+0000", 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016030100000000001 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135345+0000", 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000430438 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135847+0000", 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050190399999999996 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135873+0000", 2024-09-17T07:48:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6092e-05 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135893+0000", 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9089999999999998e-05 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:9 3.0 3:1452eb36:ns::rbd_header.12c1e94c40fb:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.882356111, 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001136375, 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 9 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1099999999999996e-07 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138081+0000", 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.926e-06 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138088+0000", 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.199e-06 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138092+0000", 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1760000000000002e-06 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138126+0000", 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4335000000000002e-05 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.139212+0000", 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010865079999999999 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.139214+0000", 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.72e-06 2024-09-17T07:48:03.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:22 3.0 3:1452eb36:ns::rbd_header.12c1e94c40fb:head [watch watch cookie 140114315468208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.86870216099999997, 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027181520000000002, 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 22 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151735+0000", 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0680000000000001e-06 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151744+0000", 2024-09-17T07:48:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910000000000003e-06 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151750+0000", 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5600000000000001e-06 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151799+0000", 2024-09-17T07:48:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.986e-05 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151883+0000", 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3824000000000002e-05 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152997+0000", 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011138509999999999 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153047+0000", 2024-09-17T07:48:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.969e-05 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153598+0000", 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055154200000000003 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153644+0000", 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5816999999999999e-05 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153718+0000", 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3388999999999998e-05 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154145+0000", 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042733700000000002 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154390+0000", 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024526099999999998 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154415+0000", 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4309999999999999e-05 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154450+0000", 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5311000000000001e-05 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.855224922, 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.84756540599999997, 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.83917583799999995, 2024-09-17T07:48:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.83880458800000002, 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:03.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:03.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.83074162299999998, 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.83037051100000003, 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:03.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.82109010599999999, 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:03.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:03.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:03.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:03.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.81943067300000005, 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.81340584800000004, 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.81315552099999999, 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.80604168300000001, 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.80552458400000004, 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.78655341700000003, 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.780296557, 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.77376049499999999, 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.147 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:03.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:03.149 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:03.148+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:03.249 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:03.249+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:03.250 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:03.250 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:03.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:03.350 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:03.350+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:03.360 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:03.360 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:03.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:03.451 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:03.451+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:7 1.7 e45/12 1:f18d6ec8:ns::rbd_id.testrbdnbd27356:head v 45'4, mlcod=45'2)", 2024-09-17T07:48:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.136642+0000", 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.378231918, 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00062958899999999995, 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136642+0000", 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136642+0000", 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136643+0000", 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136651+0000", 2024-09-17T07:48:03.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2560000000000002e-06 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136662+0000", 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1153e-05 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136668+0000", 2024-09-17T07:48:03.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2669999999999997e-06 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136703+0000", 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5540000000000002e-05 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136719+0000", 2024-09-17T07:48:03.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5787999999999999e-05 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.137235+0000", 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051565300000000003 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.137272+0000", 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7101999999999998e-05 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:29 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'12, mlcod=45'10)", 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.152809+0000", 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.3620648499999999, 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00064795800000000002, 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152809+0000", 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152809+0000", 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152810+0000", 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3799999999999996e-07 2024-09-17T07:48:03.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152817+0000", 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.742e-06 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152833+0000", 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6069000000000001e-05 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152838+0000", 2024-09-17T07:48:03.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1630000000000004e-06 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152886+0000", 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7753999999999997e-05 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152908+0000", 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2263999999999999e-05 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153422+0000", 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051435900000000004 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153457+0000", 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4869000000000001e-05 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:22 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'13, mlcod=45'11)", 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.153772+0000", 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.361101685, 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00053047700000000001, 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153772+0000", 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153772+0000", 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153773+0000", 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153778+0000", 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3970000000000002e-06 2024-09-17T07:48:03.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153790+0000", 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1773999999999999e-05 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153796+0000", 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6359999999999999e-06 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153843+0000", 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7240999999999997e-05 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153862+0000", 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9202e-05 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154276+0000", 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000413526 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154303+0000", 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6999e-05 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:24 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'14, mlcod=45'12)", 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.159072+0000", 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.3558019780000001, 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00054983500000000004, 2024-09-17T07:48:03.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159072+0000", 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159072+0000", 2024-09-17T07:48:03.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159073+0000", 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7599999999999997e-07 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159077+0000", 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1749999999999998e-06 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159087+0000", 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7350000000000008e-06 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159090+0000", 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.777e-06 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159128+0000", 2024-09-17T07:48:03.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7941999999999997e-05 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159149+0000", 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0565e-05 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159597+0000", 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00044794000000000002 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159622+0000", 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5125000000000001e-05 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.257636057, 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.536 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:62 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'15, mlcod=45'13)", 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.267646+0000", 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.247227552, 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052839300000000005, 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267646+0000", 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267646+0000", 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267647+0000", 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3699999999999999e-07 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267656+0000", 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5150000000000006e-06 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267670+0000", 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4117999999999999e-05 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267676+0000", 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4239999999999998e-06 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267712+0000", 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5117000000000003e-05 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267732+0000", 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0613999999999999e-05 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268151+0000", 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041832300000000001 2024-09-17T07:48:03.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268175+0000", 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4345000000000002e-05 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:64 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'16, mlcod=45'14)", 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.2459214759999999, 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055181100000000003, 2024-09-17T07:48:03.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:03.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8899999999999999e-07 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268960+0000", 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7920000000000004e-06 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268974+0000", 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3858e-05 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268981+0000", 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1029999999999996e-06 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269022+0000", 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1498999999999997e-05 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269043+0000", 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0259999999999999e-05 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269479+0000", 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00043659299999999999 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269504+0000", 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5017000000000001e-05 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:03.549 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:03.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:48:03.552 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:03.552+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:03.653 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:03.653+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:03.663 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:03.664 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:03.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:48:03.754 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:03.754+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:03.812 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:03.813 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:03.813 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:03.813 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:03.813 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:03.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:48:03.855 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:03.855+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:03.920 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:43 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.759252931000006, 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042712679999999999, 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 43 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159422+0000", 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159425+0000", 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9979999999999999e-06 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159433+0000", 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3629999999999996e-06 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159436+0000", 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7450000000000001e-06 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161643+0000", 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022064770000000001 2024-09-17T07:48:03.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161734+0000", 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0772000000000004e-05 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.161860+0000", 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000125907 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163606+0000", 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001746367 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163614+0000", 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2060000000000007e-06 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163657+0000", 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163693+0000", 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6109000000000003e-05 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:44 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.759184578000003, 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004857615, 2024-09-17T07:48:03.927 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 44 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159490+0000", 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159491+0000", 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159531+0000", 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0318999999999998e-05 2024-09-17T07:48:03.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159540+0000", 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2890000000000003e-06 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159544+0000", 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5130000000000001e-06 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162322+0000", 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027787459999999999 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162401+0000", 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8982999999999999e-05 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.162528+0000", 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012668799999999999 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163753+0000", 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001224876 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164271+0000", 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000517476 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164313+0000", 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2392999999999999e-05 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164348+0000", 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4765999999999999e-05 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:45 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.933 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.759126472000005, 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054853790000000003, 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 45 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159548+0000", 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159549+0000", 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7100000000000002e-07 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159551+0000", 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3199999999999998e-06 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159554+0000", 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4690000000000002e-06 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159566+0000", 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1864e-05 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163109+0000", 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035424330000000002 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163175+0000", 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6119999999999997e-05 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163301+0000", 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012651900000000001 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164962+0000", 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660171 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164966+0000", 2024-09-17T07:48:03.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8049999999999997e-06 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164998+0000", 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1300000000000002e-05 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165034+0000", 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5907e-05 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.939 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:46 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.759105090000006, 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061110779999999998, 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 46 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159570+0000", 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9700000000000003e-07 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159572+0000", 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5780000000000001e-06 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159577+0000", 2024-09-17T07:48:03.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6349999999999997e-06 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159579+0000", 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1519999999999999e-06 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163792+0000", 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042132269999999999 2024-09-17T07:48:03.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.163871+0000", 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8387999999999996e-05 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164003+0000", 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013188800000000001 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165083+0000", 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001079799 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165605+0000", 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00052247700000000003 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165646+0000", 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1075999999999998e-05 2024-09-17T07:48:03.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165681+0000", 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4561000000000003e-05 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:47 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.759090173999994, 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067770679999999998, 2024-09-17T07:48:03.946 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 47 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159584+0000", 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159585+0000", 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9400000000000001e-07 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159624+0000", 2024-09-17T07:48:03.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9066000000000002e-05 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159629+0000", 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0250000000000002e-06 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159631+0000", 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3360000000000002e-06 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164392+0000", 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047604220000000003 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164469+0000", 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7743000000000004e-05 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.164564+0000", 2024-09-17T07:48:03.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4402999999999997e-05 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166272+0000", 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017083249999999999 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166280+0000", 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7160000000000003e-06 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166325+0000", 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4824000000000003e-05 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166362+0000", 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6913999999999999e-05 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:48 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.759038524999994, 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0075774450000000004, 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.953 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159636+0000", 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159638+0000", 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0020000000000001e-06 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159644+0000", 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1420000000000001e-06 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159647+0000", 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4939999999999999e-06 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165122+0000", 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0054746020000000003 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165207+0000", 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5496999999999998e-05 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165329+0000", 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000121328 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166420+0000", 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010918340000000001 2024-09-17T07:48:03.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167129+0000", 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070897699999999996 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167179+0000", 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9551e-05 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167214+0000", 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4656999999999997e-05 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:49 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.758983009000005, 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081168149999999994, 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 49 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159692+0000", 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5600000000000005e-07 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159713+0000", 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0979e-05 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159727+0000", 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3377000000000001e-05 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159733+0000", 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1920000000000002e-06 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165726+0000", 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0059929800000000002 2024-09-17T07:48:03.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165827+0000", 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010073699999999999 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.165974+0000", 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014717199999999999 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167724+0000", 2024-09-17T07:48:03.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017499029999999999 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167732+0000", 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5609999999999999e-06 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167772+0000", 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9734000000000003e-05 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167808+0000", 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6523999999999998e-05 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:50 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.758927452999998, 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086112370000000008, 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 50 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159747+0000", 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159748+0000", 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159792+0000", 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4428999999999998e-05 2024-09-17T07:48:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159801+0000", 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8429999999999999e-06 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159806+0000", 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.51e-06 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166461+0000", 2024-09-17T07:48:03.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006655322 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166557+0000", 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5563999999999995e-05 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.166694+0000", 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013694 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167840+0000", 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011461100000000001 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168276+0000", 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043646599999999999 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168323+0000", 2024-09-17T07:48:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7042999999999997e-05 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168358+0000", 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5203999999999997e-05 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:51 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.971 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.758854751000001, 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086486050000000002, 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159820+0000", 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1699999999999999e-07 2024-09-17T07:48:03.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159824+0000", 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.23e-06 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159829+0000", 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0819999999999998e-06 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159831+0000", 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6280000000000001e-06 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167255+0000", 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0074233459999999999 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167325+0000", 2024-09-17T07:48:03.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0267999999999999e-05 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167417+0000", 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1990999999999994e-05 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168382+0000", 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096523599999999996 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168395+0000", 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2758999999999999e-05 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168434+0000", 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9498e-05 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168468+0000", 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150000000000003e-05 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:52 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.758839878000003, 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088997550000000005, 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159835+0000", 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.65e-07 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159837+0000", 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0899999999999999e-06 2024-09-17T07:48:03.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159841+0000", 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5140000000000002e-06 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.159843+0000", 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1560000000000002e-06 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167864+0000", 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080210990000000003 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.167941+0000", 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6717000000000004e-05 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168090+0000", 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014928399999999999 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168477+0000", 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038746700000000001 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168674+0000", 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019650099999999999 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168712+0000", 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7759000000000002e-05 2024-09-17T07:48:03.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.168735+0000", 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2903000000000001e-05 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430806375000003, 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044180139999999996, 2024-09-17T07:48:03.984 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 70 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487868+0000", 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487869+0000", 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.44e-07 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487874+0000", 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9459999999999997e-06 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487884+0000", 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7529999999999996e-06 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487888+0000", 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3470000000000001e-06 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490661+0000", 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027727339999999998 2024-09-17T07:48:03.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490702+0000", 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1465999999999999e-05 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.490787+0000", 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5072000000000001e-05 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491613+0000", 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082576100000000005 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492240+0000", 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062670200000000003 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492267+0000", 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7642000000000001e-05 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492286+0000", 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8947e-05 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:71 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430780210999998, 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044784200000000003, 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.990 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 71 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487894+0000", 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487895+0000", 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1600000000000001e-07 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487898+0000", 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4460000000000001e-06 2024-09-17T07:48:03.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487906+0000", 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1980000000000004e-06 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487909+0000", 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7610000000000001e-06 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491014+0000", 2024-09-17T07:48:03.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0031049419999999999 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491058+0000", 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3928999999999999e-05 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491128+0000", 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9764999999999994e-05 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492296+0000", 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001168019 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492326+0000", 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9913999999999999e-05 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492352+0000", 2024-09-17T07:48:03.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5613e-05 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492373+0000", 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1316999999999999e-05 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:72 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:48:03.996 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430759370999994, 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052882279999999999, 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:48:03.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487915+0000", 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487916+0000", 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3599999999999999e-07 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487978+0000", 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2249999999999995e-05 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487987+0000", 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487990+0000", 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3790000000000002e-06 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:03.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491639+0000", 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036484489999999998 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491689+0000", 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0015000000000001e-05 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.491792+0000", 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000103331 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493150+0000", 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013579899999999999 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493157+0000", 2024-09-17T07:48:04.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.233e-06 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493183+0000", 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5709000000000001e-05 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493203+0000", 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0455999999999999e-05 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:73 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430679127000005, 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061893290000000004, 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-09-17T07:48:04.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487996+0000", 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.53e-07 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.487998+0000", 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3769999999999999e-06 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488004+0000", 2024-09-17T07:48:04.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4319999999999998e-06 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488006+0000", 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3190000000000002e-06 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492405+0000", 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0043986069999999997 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492456+0000", 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1495000000000001e-05 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492525+0000", 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9255e-05 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494096+0000", 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015709459999999999 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494136+0000", 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9796999999999999e-05 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494163+0000", 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494185+0000", 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1557999999999999e-05 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:74 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430666145999993, 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063083990000000001, 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 74 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488008+0000", 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488009+0000", 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.89e-07 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488017+0000", 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7109999999999995e-06 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488020+0000", 2024-09-17T07:48:04.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5039999999999998e-06 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488022+0000", 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0999999999999998e-06 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492779+0000", 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0047566140000000002 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492824+0000", 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5080999999999997e-05 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.492904+0000", 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000001e-05 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494228+0000", 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001323872 2024-09-17T07:48:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494237+0000", 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9439999999999994e-06 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494282+0000", 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5441000000000001e-05 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494317+0000", 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4523000000000001e-05 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:75 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430649967999997, 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0064203569999999998, 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.015 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 75 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488025+0000", 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2999999999999999e-07 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488080+0000", 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5591999999999999e-05 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488085+0000", 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8010000000000003e-06 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488088+0000", 2024-09-17T07:48:04.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.508e-06 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493230+0000", 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051417820000000001 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493284+0000", 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4440000000000001e-05 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493351+0000", 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6760000000000005e-05 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494324+0000", 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097347500000000001 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494385+0000", 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0380000000000001e-05 2024-09-17T07:48:04.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494411+0000", 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6237999999999998e-05 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494445+0000", 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4150999999999999e-05 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:76 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.430583476999999, 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065292049999999997, 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 76 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.022 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488091+0000", 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.03e-07 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488096+0000", 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1169999999999997e-06 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488099+0000", 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2200000000000001e-06 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.488101+0000", 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.182e-06 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493768+0000", 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0056669609999999999 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493815+0000", 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7419000000000001e-05 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.493888+0000", 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2575000000000002e-05 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494454+0000", 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056606099999999997 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494576+0000", 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122359 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494603+0000", 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6197e-05 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.494620+0000", 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7810999999999999e-05 2024-09-17T07:48:04.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4186.0:94 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.016887186999995, 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042730880000000004, 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4186", 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.94:0/3828011955", 2024-09-17T07:48:04.028 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 94 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901787+0000", 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901788+0000", 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6399999999999998e-07 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901789+0000", 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.626e-06 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901792+0000", 2024-09-17T07:48:04.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1049999999999999e-06 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.901794+0000", 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.824e-06 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905170+0000", 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033753709999999998 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905239+0000", 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9663e-05 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 1", 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905359+0000", 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000120031 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905881+0000", 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000521394 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.905992+0000", 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00011144 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906032+0000", 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0287000000000002e-05 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.906061+0000", 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8082999999999999e-05 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:38 1.0 1:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.190086+0000", 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.728589036, 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.022847833000000001, 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.190086+0000", 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.190086+0000", 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.190090+0000", 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9899999999999999e-06 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206043+0000", 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015953355999999998 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206061+0000", 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8267e-05 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206067+0000", 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5740000000000003e-06 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206125+0000", 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8315e-05 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206256+0000", 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013084500000000001 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212837+0000", 2024-09-17T07:48:04.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065806279999999998 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212883+0000", 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6477999999999998e-05 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212933+0000", 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0380000000000002e-05 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.038 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:40 1.0 1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.206078+0000", 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.712596878, 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030324050000000002, 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206078+0000", 2024-09-17T07:48:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206078+0000", 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.206079+0000", 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.26e-06 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229730+0000", 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023651473999999999 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229752+0000", 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1424999999999999e-05 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229760+0000", 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7349999999999996e-06 2024-09-17T07:48:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229818+0000", 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8786999999999998e-05 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229957+0000", 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138245 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:04.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236258+0000", 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063015500000000004 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236318+0000", 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0183999999999998e-05 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236402+0000", 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3389999999999999e-05 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:04.043 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:04.044 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:04.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:04.046 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:04.044+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:04.146 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:04.146+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:04.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:04.183 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:04.183 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:04.193 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:04.247 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:04.247+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:04.291 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:04.292 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:04.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:48:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:48:04.331 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:04.331 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:04.332 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:04.332 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:04.332 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:04.338 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:04.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:04.353 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:04.353 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:04.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:04.384 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:04.384+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:04.438 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:04.438 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:04.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:04.487 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:04.486+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:04.550 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:04.550+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:04.583 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:04.583 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:04.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:04.651 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:04.651+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:04.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:48:04.692 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:04.692 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:04.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:04.752 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:04.751+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:04.840 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:04.840 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:04.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:04.892 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:04.892+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:04.951 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:04.951 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:04.953 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:04.953+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:04.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:05.055 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:05.054+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:05.101 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.102 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:05.155 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:05.155+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:05.210 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.211 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.219 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:05.256 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:05.255+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:05.321 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.321 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:05.357 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:05.356+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:05.430 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.430 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:05.482 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:05.482+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:05.540 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.540 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.549 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:48:05.595 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:05.594+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:05.647 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:48:05.653 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false rbd-nbd 2024-09-17T07:48:05.653 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd-nbd 2024-09-17T07:48:05.656 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.656 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.661 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:05.661+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:05.665 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:48:05.755 INFO:tasks.workunit.client.0.smithi098.stderr:rbd-nbd: must specify command 2024-09-17T07:48:05.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:05.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-09-17T07:48:05.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd-nbd INVALIDCMD 2024-09-17T07:48:05.759 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:05.759+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:05.774 INFO:tasks.workunit.client.0.smithi098.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-09-17T07:48:05.775 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:05.776 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:05.777 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -ne 0 ']' 2024-09-17T07:48:05.777 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd27356 2024-09-17T07:48:05.777 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd map testrbdnbd27356 2024-09-17T07:48:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.802 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:48:05.859 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:05.859+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:05.876 INFO:tasks.workunit.client.0.smithi098.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-09-17T07:48:05.878 INFO:tasks.workunit.client.0.smithi098.stderr:rbd-nbd: failed to find unused device 2024-09-17T07:48:05.883 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-17T07:48:05.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:05.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-17T07:48:05.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-17T07:48:05.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:05.887 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:05.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:05.888 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:05.890 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:05.890 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:05.890 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-09-17T07:48:05.909 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:05.910 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:05.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:05.960 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:05.960+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:05.969 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-17T07:48:05.973 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:05.973 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd27356 2024-09-17T07:48:05.973 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd27356 2024-09-17T07:48:05.973 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:05.973 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:05.974 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:05.974 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd-nbd 2024-09-17T07:48:05.976 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-17T07:48:05.976 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:05.976 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd27356 2024-09-17T07:48:06.017 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:06.017 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:06.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:06.045 INFO:tasks.workunit.client.0.smithi098.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-09-17T07:48:06.073 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:06.073+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:06.073 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:06.074 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-09-17T07:48:06.074 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd --format INVALID list 2024-09-17T07:48:06.099 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: the argument for option '--format' is invalid 2024-09-17T07:48:06.100 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:06.100 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-09-17T07:48:06.126 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:06.126 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:06.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:06.142 INFO:tasks.workunit.client.0.smithi098.stdout:[] 2024-09-17T07:48:06.144 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd --format xml list 2024-09-17T07:48:06.162 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:06.162+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:06.188 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:06.188 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:06.188 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:06.189 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:06.189 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:06.191 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:06.191 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:06.191 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:06.263 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:06.263+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:06.272 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:06.273 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:06.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:06.284 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:06.284 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:06.284 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:06.285 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:06.285 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:06.285 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:06.357 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=27692 2024-09-17T07:48:06.357 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 27692 2024-09-17T07:48:06.357 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 27692 -C rbd-nbd 2024-09-17T07:48:06.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:06.365 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:06.365+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:06.371 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:06.371 INFO:tasks.workunit.client.0.smithi098.stdout: 27692 ? 00:00:00 rbd-nbd 2024-09-17T07:48:06.371 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27356 2024-09-17T07:48:06.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27356 2024-09-17T07:48:06.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:06.372 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:06.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:06.373 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd-nbd 2024-09-17T07:48:06.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-17T07:48:06.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:06.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27356 2024-09-17T07:48:06.420 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:06.420 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:06.429 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:06.438 INFO:tasks.workunit.client.0.smithi098.stderr:rbd-nbd: the device /dev/nbd0 is busy 2024-09-17T07:48:06.445 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:06.445 INFO:tasks.workunit.client.0.smithi098.stderr:+ dev1=/dev/nbd0 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 27692 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=27692 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:06.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:06.447 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:06.448 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:06.448 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:06.448 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:06.465 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:06.465+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:06.466 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:06.482 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:06.482 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:06.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:06.512 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:06.512 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^27692\b' 2024-09-17T07:48:06.512 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^27692\b' 2024-09-17T07:48:06.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:48:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:48:06.530 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:06.530 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:06.547 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:06.547 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:06.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:06.551 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:06.551 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:06.551 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *27692\b' 2024-09-17T07:48:06.552 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *27692\b' 2024-09-17T07:48:06.561 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:06.561 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:06.566 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:06.566+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:06.568 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:06.568 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:06.569 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:06.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:06.667 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:06.667+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.714105266000004, 2024-09-17T07:48:06.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:48:06.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:48:06.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.5335230050000002, 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:06.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.725 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.5253829669999996, 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.5057602750000001, 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4984350429999997, 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:06.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:06.740 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4918229419999998, 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:06.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:06.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:06.745 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4876416189999997, 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:06.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:06.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4853289749999998, 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:06.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:06.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:06.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4602818710000003, 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:06.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:06.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:06.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4599444139999997, 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:06.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4801.0:60 1.7 e45/12 ondisk, result = 0)", 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.4494322039999998, 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012042, 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5099999999999999e-07 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265799+0000", 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4870000000000004e-06 2024-09-17T07:48:06.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265809+0000", 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0304e-05 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265814+0000", 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3959999999999999e-06 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265845+0000", 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1043000000000001e-05 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265850+0000", 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7150000000000003e-06 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265914+0000", 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3724000000000002e-05 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4819.0:1 4.3 4:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:03.642188+0000", 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.073037759, 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00073594700000000003, 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4819", 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2614708871", 2024-09-17T07:48:06.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642188+0000", 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642188+0000", 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642190+0000", 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.562e-06 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642195+0000", 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5990000000000001e-06 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:06.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642207+0000", 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1761e-05 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642216+0000", 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4979999999999995e-06 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642258+0000", 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1662000000000002e-05 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642369+0000", 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111366 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642806+0000", 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043668100000000002 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642852+0000", 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5510000000000003e-05 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:06.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642924+0000", 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2308e-05 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:06.775 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:06.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:06.777 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:06.776+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:06.878 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:06.877+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:06.879 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:06.879 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:06.879 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:06.879 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:06.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:06.978 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:06.978+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:06.988 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:06.989 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:06.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:07.062 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:07.063 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27356 2024-09-17T07:48:07.063 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:07.063 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:07.064 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:07.064 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd-nbd 2024-09-17T07:48:07.066 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd-nbd 2024-09-17T07:48:07.066 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:07.066 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27356 2024-09-17T07:48:07.079 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:07.079+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4187.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:48:07.100 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.097899405000007, 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013222069999999999, 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4187", 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.26:0/2715881128", 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 1 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000583+0000", 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000585+0000", 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.714e-06 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000593+0000", 2024-09-17T07:48:07.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.627e-06 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000609+0000", 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6560000000000001e-05 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000622+0000", 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.275e-05 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000686+0000", 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3882999999999998e-05 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000809+0000", 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012330800000000001 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.000920+0000", 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111209 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001476+0000", 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000555858 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001836+0000", 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036008500000000001 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001868+0000", 2024-09-17T07:48:07.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1973e-05 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.001905+0000", 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7240000000000003e-05 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4186.0:21 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.081448391999999, 2024-09-17T07:48:07.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011487240000000001, 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017034+0000", 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017035+0000", 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6600000000000005e-07 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017136+0000", 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100899 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017149+0000", 2024-09-17T07:48:07.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2917e-05 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017153+0000", 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017192+0000", 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8986000000000002e-05 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.017206+0000", 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4021e-05 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018156+0000", 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094916300000000005 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:46:53.018183+0000", 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7368999999999998e-05 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:6 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_assert in=19b,call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.963811916, 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001221846, 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 6 2024-09-17T07:48:07.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134671+0000", 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6300000000000005e-07 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134674+0000", 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0989999999999999e-06 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134681+0000", 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1179999999999997e-06 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134683+0000", 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8449999999999999e-06 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134706+0000", 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2849000000000001e-05 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134755+0000", 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8547000000000001e-05 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.134915+0000", 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016030100000000001 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135345+0000", 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000430438 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135847+0000", 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00050190399999999996 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135873+0000", 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6092e-05 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.135893+0000", 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9089999999999998e-05 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.116 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:9 3.0 3:1452eb36:ns::rbd_header.12c1e94c40fb:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9604049230000005, 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001136375, 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 9 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:07.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138078+0000", 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1099999999999996e-07 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138081+0000", 2024-09-17T07:48:07.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.926e-06 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138088+0000", 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.199e-06 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138092+0000", 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1760000000000002e-06 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.138126+0000", 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4335000000000002e-05 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.139212+0000", 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010865079999999999 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.139214+0000", 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.72e-06 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:22 3.0 3:1452eb36:ns::rbd_header.12c1e94c40fb:head [watch watch cookie 140114315468208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9467509730000003, 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027181520000000002, 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 22 2024-09-17T07:48:07.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151732+0000", 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151735+0000", 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0680000000000001e-06 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151744+0000", 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910000000000003e-06 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151750+0000", 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5600000000000001e-06 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151799+0000", 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.986e-05 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.151883+0000", 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3824000000000002e-05 2024-09-17T07:48:07.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152997+0000", 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011138509999999999 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153047+0000", 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.969e-05 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153598+0000", 2024-09-17T07:48:07.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055154200000000003 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153644+0000", 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5816999999999999e-05 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153718+0000", 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3388999999999998e-05 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154145+0000", 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042733700000000002 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154390+0000", 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024526099999999998 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154415+0000", 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4309999999999999e-05 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154450+0000", 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5311000000000001e-05 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9332737340000001, 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9256142179999998, 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:07.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.137 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9172246499999996, 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:07.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9168533999999999, 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.143 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.147 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9087904350000002, 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.9084193230000004, 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:07.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8991389180000002, 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:07.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8974794849999999, 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.89145466, 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:07.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8912043330000001, 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8840904949999997, 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:07.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:07.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8835733960000001, 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:07.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:07.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.186 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8646022289999999, 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:07.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8583453690000002, 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:07.191 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:07.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:07.195 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8518093069999999, 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:07.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:07.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:07.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.201 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:07.202 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:07.202+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:07.203 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:07.203 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-09-17T07:48:07.203 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:07.203 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep testrbdnbd27356 2024-09-17T07:48:07.204 INFO:tasks.workunit.client.0.smithi098.stdout:27795 rbd testrbdnbd27356 - /dev/nbd0 2024-09-17T07:48:07.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:07.205 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:07.205 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:07.205 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:07.205 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:07.205 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:07.242 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=27795 2024-09-17T07:48:07.242 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 27795 2024-09-17T07:48:07.243 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 27795 -C rbd-nbd 2024-09-17T07:48:07.253 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:07.253 INFO:tasks.workunit.client.0.smithi098.stdout: 27795 ? 00:00:00 rbd-nbd 2024-09-17T07:48:07.253 INFO:tasks.workunit.client.0.smithi098.stderr:++ dd if=/tmp/tmp.TyfTMF4aW5/data bs=1M 2024-09-17T07:48:07.253 INFO:tasks.workunit.client.0.smithi098.stderr:++ md5sum 2024-09-17T07:48:07.302 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:07.302+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:07.306 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:07.307 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:07.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:07.367 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:07.367 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:07.367 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.112875 s, 595 MB/s 2024-09-17T07:48:07.368 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-09-17T07:48:07.368 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:07.368 INFO:tasks.workunit.client.0.smithi098.stderr:++ md5sum 2024-09-17T07:48:07.368 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:07.369 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:07.369 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which dd 2024-09-17T07:48:07.371 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/dd 2024-09-17T07:48:07.371 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:07.371 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-09-17T07:48:07.403 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:07.403+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:07.419 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:07.419 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:07.419 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:07.419 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:07.420 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:07.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:07.504 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:07.504+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:7 1.7 e45/12 1:f18d6ec8:ns::rbd_id.testrbdnbd27356:head v 45'4, mlcod=45'2)", 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.136642+0000", 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.4019296240000001, 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00062958899999999995, 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136642+0000", 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136642+0000", 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136643+0000", 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2999999999999999e-07 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136651+0000", 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2560000000000002e-06 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136662+0000", 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1153e-05 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136668+0000", 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2669999999999997e-06 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136703+0000", 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5540000000000002e-05 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.136719+0000", 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5787999999999999e-05 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.137235+0000", 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051565300000000003 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.137272+0000", 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7101999999999998e-05 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:29 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'12, mlcod=45'10)", 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.152809+0000", 2024-09-17T07:48:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.3857625559999995, 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00064795800000000002, 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152809+0000", 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152809+0000", 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152810+0000", 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3799999999999996e-07 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152817+0000", 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.742e-06 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152833+0000", 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6069000000000001e-05 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152838+0000", 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1630000000000004e-06 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152886+0000", 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7753999999999997e-05 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.152908+0000", 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2263999999999999e-05 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153422+0000", 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051435900000000004 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153457+0000", 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4869000000000001e-05 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:22 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'13, mlcod=45'11)", 2024-09-17T07:48:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.153772+0000", 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.3847993909999996, 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00053047700000000001, 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153772+0000", 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153772+0000", 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153773+0000", 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153778+0000", 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3970000000000002e-06 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153790+0000", 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1773999999999999e-05 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153796+0000", 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6359999999999999e-06 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153843+0000", 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7240999999999997e-05 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.153862+0000", 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9202e-05 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154276+0000", 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000413526 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.154303+0000", 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6999e-05 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:24 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'14, mlcod=45'12)", 2024-09-17T07:48:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.159072+0000", 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.3794996839999998, 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00054983500000000004, 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159072+0000", 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159072+0000", 2024-09-17T07:48:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159073+0000", 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7599999999999997e-07 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159077+0000", 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1749999999999998e-06 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159087+0000", 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7350000000000008e-06 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159090+0000", 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.777e-06 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159128+0000", 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7941999999999997e-05 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159149+0000", 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0565e-05 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159597+0000", 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00044794000000000002 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.159622+0000", 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5125000000000001e-05 2024-09-17T07:48:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.2813337630000001, 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:62 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'15, mlcod=45'13)", 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.267646+0000", 2024-09-17T07:48:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.2709252580000001, 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052839300000000005, 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267646+0000", 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267646+0000", 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267647+0000", 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3699999999999999e-07 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267656+0000", 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5150000000000006e-06 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267670+0000", 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4117999999999999e-05 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267676+0000", 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4239999999999998e-06 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267712+0000", 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5117000000000003e-05 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.267732+0000", 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0613999999999999e-05 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268151+0000", 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041832300000000001 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268175+0000", 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4345000000000002e-05 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:64 3.0 e45/16 3:1452eb36:ns::rbd_header.12c1e94c40fb:head v 45'16, mlcod=45'14)", 2024-09-17T07:48:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.2696191819999996, 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055181100000000003, 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268953+0000", 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8899999999999999e-07 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268960+0000", 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7920000000000004e-06 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268974+0000", 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3858e-05 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.268981+0000", 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1029999999999996e-06 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269022+0000", 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1498999999999997e-05 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269043+0000", 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0259999999999999e-05 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269479+0000", 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00043659299999999999 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.269504+0000", 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5017000000000001e-05 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4879.0:118 3.0 e49/16 3:0cbfebbd:::rbd_header.123eb9c2b614:head v 49'25, mlcod=49'23)", 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:06.505262+0000", 2024-09-17T07:48:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.033309443, 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049119400000000003, 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505262+0000", 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505262+0000", 2024-09-17T07:48:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505263+0000", 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505269+0000", 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3520000000000003e-06 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505280+0000", 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0560999999999999e-05 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505286+0000", 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.755e-06 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505316+0000", 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0608000000000002e-05 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505339+0000", 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2844e-05 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505726+0000", 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00038694499999999998 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:06.505753+0000", 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7520000000000001e-05 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:12 3.0 e49/16 3:0cbfebbd:::rbd_header.123eb9c2b614:head v 49'26, mlcod=49'24)", 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.124548+0000", 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.41402364699999999, 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0006154, 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124548+0000", 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124548+0000", 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124550+0000", 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.598e-06 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124560+0000", 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0003999999999999e-05 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124580+0000", 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0009000000000001e-05 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124588+0000", 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3690000000000007e-06 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124629+0000", 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0670999999999997e-05 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.124647+0000", 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8522e-05 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.125136+0000", 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00048902799999999997 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.125163+0000", 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7199000000000002e-05 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:160 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.436958+0000", 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.10161379700000001, 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072366499999999996, 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 160 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.436958+0000", 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.436958+0000", 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.436958+0000", 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2e-07 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.436960+0000", 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1100000000000001e-06 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.436966+0000", 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8880000000000002e-06 2024-09-17T07:48:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.436971+0000", 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2570000000000001e-06 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.437306+0000", 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033532100000000002 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.437364+0000", 2024-09-17T07:48:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7717999999999999e-05 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.437682+0000", 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031795100000000002 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:162 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.439076+0000", 2024-09-17T07:48:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.099496197999999994, 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079780699999999999, 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 162 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439076+0000", 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439076+0000", 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439076+0000", 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0900000000000002e-07 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439079+0000", 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9409999999999999e-06 2024-09-17T07:48:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439087+0000", 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0870000000000007e-06 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439092+0000", 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9099999999999996e-06 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439495+0000", 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00040339899999999998 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439547+0000", 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1425000000000003e-05 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.439873+0000", 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032663599999999999 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:166 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 790528~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.440896+0000", 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.097676123000000004, 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00060181299999999998, 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 166 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.440896+0000", 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.440896+0000", 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.440896+0000", 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.440899+0000", 2024-09-17T07:48:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7819999999999998e-06 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.440907+0000", 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4419999999999998e-06 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.440913+0000", 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6860000000000003e-06 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441117+0000", 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000204255 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441197+0000", 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-05 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441497+0000", 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030039900000000002 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:167 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.441471+0000", 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.097100619999999999, 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049579300000000001, 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 167 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441471+0000", 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441471+0000", 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441472+0000", 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5199999999999996e-07 2024-09-17T07:48:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441476+0000", 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1670000000000001e-06 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441485+0000", 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2580000000000001e-06 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441489+0000", 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1910000000000002e-06 2024-09-17T07:48:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441526+0000", 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6628999999999998e-05 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441615+0000", 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8968999999999995e-05 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.441967+0000", 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00035182699999999998 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:169 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 1183744~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.443922+0000", 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.094649621000000003, 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00053976999999999996, 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 169 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443922+0000", 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443922+0000", 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443923+0000", 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2699999999999999e-07 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443926+0000", 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5880000000000002e-06 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443937+0000", 2024-09-17T07:48:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0414999999999999e-05 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443941+0000", 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2570000000000001e-06 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443981+0000", 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9802000000000003e-05 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.444074+0000", 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3374000000000004e-05 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.444462+0000", 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00038760699999999999 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:170 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.443946+0000", 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.094625253000000006, 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00089119500000000001, 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 170 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443946+0000", 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443946+0000", 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443947+0000", 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8599999999999999e-07 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443948+0000", 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.31e-06 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443951+0000", 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4240000000000002e-06 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.443954+0000", 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1220000000000002e-06 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.444495+0000", 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00054186000000000002 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.444558+0000", 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2180000000000004e-05 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.444838+0000", 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00028001300000000002 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:172 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 1576960~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.445167+0000", 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.093404977, 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052663700000000005, 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 172 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445167+0000", 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445167+0000", 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445168+0000", 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.85e-07 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445172+0000", 2024-09-17T07:48:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1409999999999998e-06 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445181+0000", 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5389999999999996e-06 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445186+0000", 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7489999999999998e-06 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445259+0000", 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3337999999999995e-05 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445355+0000", 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5753000000000003e-05 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.445693+0000", 2024-09-17T07:48:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000338332 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:178 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.448978+0000", 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.089594069999999998, 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067975100000000003, 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 178 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.448978+0000", 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.448978+0000", 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.448978+0000", 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8999999999999997e-07 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.448981+0000", 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5059999999999998e-06 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.448988+0000", 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8339999999999999e-06 2024-09-17T07:48:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.448992+0000", 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9380000000000002e-06 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.449337+0000", 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00034480200000000001 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.449384+0000", 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6771999999999999e-05 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.449657+0000", 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00027340900000000001 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:184 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 3149824~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.451443+0000", 2024-09-17T07:48:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.087128273000000006, 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000526138, 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 184 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451443+0000", 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451443+0000", 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451444+0000", 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451447+0000", 2024-09-17T07:48:07.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8329999999999998e-06 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451455+0000", 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9330000000000001e-06 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451459+0000", 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6809999999999997e-06 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451601+0000", 2024-09-17T07:48:07.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000141328 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451649+0000", 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8047999999999998e-05 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.451970+0000", 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000320814 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:186 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.453541+0000", 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.085030600999999997, 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082086400000000003, 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 186 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453541+0000", 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453541+0000", 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453541+0000", 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7e-07 2024-09-17T07:48:07.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453543+0000", 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.857e-06 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453549+0000", 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0929999999999999e-06 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453563+0000", 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3694e-05 2024-09-17T07:48:07.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.453968+0000", 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000404709 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454065+0000", 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7039000000000004e-05 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454362+0000", 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00029710199999999999 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:188 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [sparse-read 3674112~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.454716+0000", 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.083855529999999998, 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051021300000000003, 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:07.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 188 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454716+0000", 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454716+0000", 2024-09-17T07:48:07.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454717+0000", 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3900000000000004e-07 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454721+0000", 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9580000000000001e-06 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:07.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454730+0000", 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3850000000000008e-06 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454735+0000", 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0159999999999999e-06 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454810+0000", 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4665000000000004e-05 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:07.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.454904+0000", 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4231e-05 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.455226+0000", 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000322119 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:07.626 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:07.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:48:07.628 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:48:07.627+0000 7f488a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31597) UID: 0 2024-09-17T07:48:07.655 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5] 2024-09-17T07:48:07.655 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:07.655 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 2, 3, 4, 5] 2024-09-17T07:48:07.655 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T07:48:07.655 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:48:07.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:48:07.701 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T07:48:07.701 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-09-17T07:48:07.701 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:48:07.701 INFO:tasks.thrashosds.thrasher:remote for exp osd 0 is ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:48:07.702 DEBUG:teuthology.orchestra.run.smithi026:> type ceph-objectstore-tool 2024-09-17T07:48:07.724 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-17T07:48:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:48:07.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:48:07.728 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T07:48:07.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-17T07:48:07.729 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:07.729 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:07.729 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.349509 s, 192 MB/s 2024-09-17T07:48:07.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 'ddd6a869765b165bfff67dc94a2a4630 -' = 'ddd6a869765b165bfff67dc94a2a4630 -' ']' 2024-09-17T07:48:07.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/urandom of=/tmp/tmp.TyfTMF4aW5/data bs=1M count=64 2024-09-17T07:48:07.768 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:07.768+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:07.829 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:07.829+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:07.852 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-17T07:48:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:48:07.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:48:07.930 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:07.930+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:07.940 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-17T07:48:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:48:07.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:08.031 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:08.031+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:08.040 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:08.040 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:08.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:08.132 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:08.132+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:08.186 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:08.187 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:08.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:08.231 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:08.231 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:08.231 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.491831 s, 136 MB/s 2024-09-17T07:48:08.232 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:08.232 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:08.232 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:08.233 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:08.233+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:08.233 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:08.233 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:48:08.235 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:48:08.235 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:08.235 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:08.299 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4187.0:1 1.3 e13/12 1:c4c92e5a:::rbd_trash:head v 13'1, mlcod=0'0)", 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.296815581999994, 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00060739400000000003, 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.300 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001121+0000", 2024-09-17T07:48:08.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6100000000000002e-07 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001126+0000", 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3669999999999996e-06 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001138+0000", 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1752e-05 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001146+0000", 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2199999999999992e-06 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001187+0000", 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0439999999999999e-05 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001211+0000", 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4341999999999998e-05 2024-09-17T07:48:08.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001686+0000", 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000475289 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:46:53.001728+0000", 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1622999999999998e-05 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.1162333210000002, 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:08.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.1080932829999997, 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.309 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0884705910000001, 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:08.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.319 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0811453589999998, 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.323 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0745332579999998, 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:08.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0703519349999997, 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:08.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:08.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0680392909999998, 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.333 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:08.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:08.338 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0429921870000003, 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:08.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:08.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0426547299999998, 2024-09-17T07:48:08.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:08.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop_reply(client.4801.0:60 1.7 e45/12 ondisk, result = 0)", 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0321425199999998, 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012042, 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:08.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265794+0000", 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5099999999999999e-07 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265799+0000", 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4870000000000004e-06 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265809+0000", 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0304e-05 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265814+0000", 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3959999999999999e-06 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265845+0000", 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1043000000000001e-05 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265850+0000", 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7150000000000003e-06 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265914+0000", 2024-09-17T07:48:08.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3724000000000002e-05 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4819.0:1 4.3 4:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:03.642188+0000", 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.655748075, 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00073594700000000003, 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4819", 2024-09-17T07:48:08.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2614708871", 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642188+0000", 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642188+0000", 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642190+0000", 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.562e-06 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642195+0000", 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5990000000000001e-06 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642207+0000", 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1761e-05 2024-09-17T07:48:08.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642216+0000", 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4979999999999995e-06 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642258+0000", 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1662000000000002e-05 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642369+0000", 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000111366 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642806+0000", 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00043668100000000002 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642852+0000", 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5510000000000003e-05 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:03.642924+0000", 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2308e-05 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.358 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:54 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [read 0~4096] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588757+0000", 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.70917898499999998, 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025765800000000002, 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588757+0000", 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588757+0000", 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588759+0000", 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5069999999999999e-06 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588766+0000", 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5349999999999999e-06 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588781+0000", 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4980999999999999e-05 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588788+0000", 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6579999999999996e-06 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588832+0000", 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3789999999999999e-05 2024-09-17T07:48:08.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588889+0000", 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6950000000000002e-05 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589015+0000", 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012623699999999999 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:55 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588797+0000", 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.70913907399999998, 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00034791100000000001, 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588797+0000", 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588797+0000", 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588797+0000", 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2599999999999998e-07 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:08.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3259999999999998e-06 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588806+0000", 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3690000000000003e-06 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588811+0000", 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5390000000000001e-06 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589050+0000", 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000239373 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589096+0000", 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6399000000000001e-05 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589145+0000", 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8479000000000001e-05 2024-09-17T07:48:08.366 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.70911130200000005, 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:08.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:60 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [read 0~4096] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682659+0000", 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.61527688300000005, 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00034777700000000001, 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682659+0000", 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682659+0000", 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682661+0000", 2024-09-17T07:48:08.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9209999999999999e-06 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682669+0000", 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9039999999999992e-06 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682690+0000", 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0594e-05 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682698+0000", 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0059999999999992e-06 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682813+0000", 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000115505 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682885+0000", 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1947999999999996e-05 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683007+0000", 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012189900000000001 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:61 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682774+0000", 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.61516243299999995, 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062398399999999995, 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.375 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682774+0000", 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682774+0000", 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682774+0000", 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3e-07 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682778+0000", 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5839999999999999e-06 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682787+0000", 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0049999999999993e-06 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682816+0000", 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9169000000000001e-05 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683031+0000", 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021445700000000001 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683106+0000", 2024-09-17T07:48:08.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5679000000000005e-05 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683398+0000", 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029146 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:62 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682828+0000", 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.61510839100000003, 2024-09-17T07:48:08.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00067536099999999997, 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682828+0000", 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682828+0000", 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682828+0000", 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7700000000000004e-07 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682831+0000", 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8710000000000001e-06 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682838+0000", 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9589999999999998e-06 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682842+0000", 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9709999999999998e-06 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683414+0000", 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000572129 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683459+0000", 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5089000000000001e-05 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683503+0000", 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3764999999999999e-05 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.383 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.61507930600000005, 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:08.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:08.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.388 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:08.388 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:08.388 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:08.389 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:08.389+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:08.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:08.490 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:08.490+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:08.501 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:08.502 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:08.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:08.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:08.589 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:08.589 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:08.589 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.342618 s, 196 MB/s 2024-09-17T07:48:08.591 INFO:tasks.workunit.client.0.smithi098.stderr:++ dd if=/tmp/tmp.TyfTMF4aW5/data bs=1M 2024-09-17T07:48:08.592 INFO:tasks.workunit.client.0.smithi098.stderr:++ md5sum 2024-09-17T07:48:08.598 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:08.598+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:08.619 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:08.620 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:08.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:48:08.668 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:08.684 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:08.685 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:08.692 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:08.692+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:08.697 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:08.697 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:08.697 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.105239 s, 638 MB/s 2024-09-17T07:48:08.698 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --no-progress export testrbdnbd27356 - 2024-09-17T07:48:08.698 INFO:tasks.workunit.client.0.smithi098.stderr:++ md5sum 2024-09-17T07:48:08.701 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:08.702 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:08.702 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:08.717 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:08.717 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:08.733 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:08.734 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:08.738 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:08.751 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:48:08.751+0000 7f25c9b4e640 0 -- 172.21.15.98:0/1429037093 >> [v2:172.21.15.94:6808/31605,v1:172.21.15.94:6809/31605] conn(0x56233115de50 msgr2=0x562331160230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:48:08.755 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:48:08.755+0000 7f25c934d640 0 -- 172.21.15.98:0/1429037093 >> [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] conn(0x7f25a4008980 msgr2=0x7f25a400ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:48:08.763 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:08.763 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:08.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:08.815 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:08.815+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:08.872 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 'a02ae039d867a200122082f7464653ce -' = 'a02ae039d867a200122082f7464653ce -' ']' 2024-09-17T07:48:08.872 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 27795 2024-09-17T07:48:08.872 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:08.873 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=27795 2024-09-17T07:48:08.873 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:08.873 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:08.873 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:08.873 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:08.873 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:08.875 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:08.875 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:08.875 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:08.878 INFO:teuthology.orchestra.run.smithi026.stdout:3.3 2024-09-17T07:48:08.878 INFO:teuthology.orchestra.run.smithi026.stdout:3.1 2024-09-17T07:48:08.878 INFO:teuthology.orchestra.run.smithi026.stdout:1.6 2024-09-17T07:48:08.878 INFO:teuthology.orchestra.run.smithi026.stdout:1.5 2024-09-17T07:48:08.878 INFO:teuthology.orchestra.run.smithi026.stdout:1.2 2024-09-17T07:48:08.879 INFO:teuthology.orchestra.run.smithi026.stdout:1.1 2024-09-17T07:48:08.879 INFO:teuthology.orchestra.run.smithi026.stdout:1.0 2024-09-17T07:48:08.894 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:08.894+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:08.926 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:08.927 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:08.935 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:08.949 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:08.949 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^27795\b' 2024-09-17T07:48:08.949 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^27795\b' 2024-09-17T07:48:08.987 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:08.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:08.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *27795\b' 2024-09-17T07:48:08.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *27795\b' 2024-09-17T07:48:08.995 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:08.995+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:08.998 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:08.998 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:09.036 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.036 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:09.096 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:09.096+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:09.146 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.146 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:09.202 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:09.202+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:09.256 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.256 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:09.309 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:09.305+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:09.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 1.6 --file /var/log/ceph/exp.1.6.0 2024-09-17T07:48:09.398 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:09.398+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:09.400 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.401 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:09.499 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:09.498+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:09.499 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd27356 2024-09-17T07:48:09.499 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:09.500 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:09.500 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:09.501 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:09.503 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:09.503 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:09.503 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd27356 2024-09-17T07:48:09.547 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.547 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:09.601 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:09.602 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:09.602 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:09.602 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:09.602 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:09.602 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:09.639 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:09.639+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:09.657 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=27914 2024-09-17T07:48:09.657 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 27914 2024-09-17T07:48:09.657 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 27914 -C rbd-nbd 2024-09-17T07:48:09.667 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:09.667 INFO:tasks.workunit.client.0.smithi098.stdout: 27914 ? 00:00:00 rbd-nbd 2024-09-17T07:48:09.668 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:09.668 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-17T07:48:09.693 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.693 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.701 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:09.700+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:09.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:09.709 INFO:tasks.workunit.client.0.smithi098.stderr:+ provisioned=67108864 2024-09-17T07:48:09.709 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:09.709 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-17T07:48:09.750 INFO:tasks.workunit.client.0.smithi098.stderr:+ used=67108864 2024-09-17T07:48:09.750 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-17T07:48:09.751 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-09-17T07:48:09.751 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-17T07:48:09.751 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:09.751 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:09.751 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:09.751 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:48:09.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:48:09.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:09.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-17T07:48:09.801 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:09.801+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:09.817 INFO:tasks.workunit.client.0.smithi098.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-09-17T07:48:09.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:09.819 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-09-17T07:48:09.840 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.840 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:09.933 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:09.933+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:09.986 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:09.987 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:09.995 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:10.040 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:10.040+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:10.097 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.097 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.105 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:10.104+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:10.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:10.205 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:10.205+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:10.248 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.248 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:10.306 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:10.306+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:10.398 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.399 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:10.408 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:10.406+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:10.496 INFO:teuthology.orchestra.run.smithi026.stderr:Exporting 1.6 info 1.6( empty local-lis/les=29/30 n=0 ec=12/12 lis/c=29/29 les/c/f=30/30/0 sis=29) 2024-09-17T07:48:10.497 INFO:teuthology.orchestra.run.smithi026.stderr:Export successful 2024-09-17T07:48:10.507 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:10.507+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:10.551 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.552 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:10.608 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:10.607+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:10.662 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.662 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:48:10.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:10.719 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:10.717+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:10.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:10.856 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.857 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:10.858 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:10.856+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:10.956 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:10.956+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:10.967 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:10.967 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:10.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:10.997 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:11.013 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:11.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:11.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 1.6 2024-09-17T07:48:11.030 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:11.030 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:11.030 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:11.047 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:11.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:11.060 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:11.060+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:11.063 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:11.063 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:11.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:11.115 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:11.115 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:11.115 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:11.116 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:11.116 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:11.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:11.158 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:11.158+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:11.259 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:11.258+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0953072820000003, 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:11.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0879679479999993, 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:11.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:11.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0872508740000004, 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:11.271 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:11.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0780233129999992, 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.276 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:11.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:11.280 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0687987450000005, 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:11.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.284 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0533241049999997, 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:11.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0440345949999994, 2024-09-17T07:48:11.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:11.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:11.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0374443499999995, 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.294 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:11.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:53 3.1 3:880ad092:ns::rbd_object_map.12c1e94c40fb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229491+0000", 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0310420140000005, 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037877589999999999, 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.298 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 53 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229491+0000", 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229491+0000", 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229493+0000", 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5629999999999999e-06 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229498+0000", 2024-09-17T07:48:11.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9860000000000002e-06 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229526+0000", 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8147e-05 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229539+0000", 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3684e-05 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229949+0000", 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00040913400000000002 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.230077+0000", 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012872 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233132+0000", 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003054672 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233194+0000", 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2360999999999995e-05 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233279+0000", 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4492000000000004e-05 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:54 1.6 1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.238591+0000", 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0219424749999995, 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011891762, 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.238591+0000", 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.238591+0000", 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.238593+0000", 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.013e-06 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244939+0000", 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063460770000000003 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244950+0000", 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1109e-05 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244956+0000", 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6499999999999999e-06 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245010+0000", 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3397999999999999e-05 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245155+0000", 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014553100000000001 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.250301+0000", 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005145635 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.250384+0000", 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2758999999999998e-05 2024-09-17T07:48:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.250482+0000", 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8590000000000003e-05 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0212618689999999, 2024-09-17T07:48:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:58 1.2 1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.247046+0000", 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0134870659999997, 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011059459000000001, 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.247046+0000", 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.247046+0000", 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.247047+0000", 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2929999999999999e-06 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.253158+0000", 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0061103650000000004 2024-09-17T07:48:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.253179+0000", 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1798e-05 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.253190+0000", 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0929e-05 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.253234+0000", 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3930000000000001e-05 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.253342+0000", 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000107509 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257971+0000", 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0046287400000000001 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.258032+0000", 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1444000000000003e-05 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.258105+0000", 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3450999999999999e-05 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0121869490000002, 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:11.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.6717480729999998, 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:11.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.6696957960000001, 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.609464145, 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:491 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.653245+0000", 2024-09-17T07:48:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.6072876699999998, 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011885903, 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 491 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653245+0000", 2024-09-17T07:48:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653245+0000", 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653246+0000", 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653250+0000", 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8519999999999997e-06 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653259+0000", 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7679999999999997e-06 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653264+0000", 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3730000000000001e-06 2024-09-17T07:48:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653293+0000", 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8467e-05 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.653320+0000", 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7827999999999999e-05 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662778+0000", 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0094579950000000003 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662871+0000", 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2188999999999994e-05 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664727+0000", 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018567449999999999 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664812+0000", 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4637999999999998e-05 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.665131+0000", 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00031941599999999999 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:524 3.3 3:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.685057+0000", 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.5754761359999998, 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090074720000000007, 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 524 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685057+0000", 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685057+0000", 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685058+0000", 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.147e-06 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685064+0000", 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6969999999999998e-06 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685076+0000", 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1980999999999999e-05 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685082+0000", 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7039999999999997e-06 2024-09-17T07:48:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685132+0000", 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9336999999999999e-05 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685185+0000", 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2689000000000003e-05 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.691729+0000", 2024-09-17T07:48:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.006544879 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.691836+0000", 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000106654 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.693574+0000", 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001738213 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.693667+0000", 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2527000000000004e-05 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.694064+0000", 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000397644 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:525 3.3 3:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.685091+0000", 2024-09-17T07:48:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.5754421079999998, 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0094867839999999998, 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 525 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685091+0000", 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685091+0000", 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685091+0000", 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9900000000000001e-07 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685094+0000", 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-06 2024-09-17T07:48:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685100+0000", 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1820000000000003e-06 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685105+0000", 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.955e-06 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685205+0000", 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000100148 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.685233+0000", 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8092000000000001e-05 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.691864+0000", 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066312389999999997 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.691927+0000", 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2231999999999996e-05 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.694190+0000", 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002263737 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.694278+0000", 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7739999999999997e-05 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.694578+0000", 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00029965999999999999 2024-09-17T07:48:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:594 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.252873+0000", 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.007660483, 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035335240000000001, 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 594 2024-09-17T07:48:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.252873+0000", 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.252873+0000", 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.252873+0000", 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4200000000000002e-07 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.252953+0000", 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9863000000000004e-05 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.252961+0000", 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2160000000000006e-06 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.252964+0000", 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.694e-06 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.254400+0000", 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001436253 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.254459+0000", 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8539000000000003e-05 2024-09-17T07:48:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.256331+0000", 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018727640000000001 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.256358+0000", 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6754000000000002e-05 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.256406+0000", 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8099000000000001e-05 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.365 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:11.366 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:11.367 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:11.366+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:11.466 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:11.466 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:11.466 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:11.466 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:11.467 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:11.467+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:11.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:11.570 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:11.570+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:11.612 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:11.612 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:11.612 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:11.612 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:11.612 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:11.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:11.634 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:11.634 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-17T07:48:11.655 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:48:11.670 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:11.670+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:11.682 INFO:tasks.workunit.client.0.smithi098.stderr:+ provisioned=67108864 2024-09-17T07:48:11.683 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:11.683 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.5400801120000001, 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:11.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.5319400739999995, 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:11.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:11.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.5123173819999991, 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.5049921499999996, 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.4983800489999997, 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.4941987260000005, 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.4918860820000006, 2024-09-17T07:48:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.4668389780000002, 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.4665015209999996, 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.1329580930000001, 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.038926097, 2024-09-17T07:48:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:611 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'8, mlcod=49'6)", 2024-09-17T07:48:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.272320+0000", 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.449462805, 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010773099999999999, 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272320+0000", 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272320+0000", 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272321+0000", 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2270000000000001e-06 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272488+0000", 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016690100000000001 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272503+0000", 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4817999999999999e-05 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272510+0000", 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5769999999999999e-06 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272555+0000", 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5519999999999998e-05 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.272577+0000", 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1461999999999999e-05 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.273353+0000", 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00077597799999999998 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.273397+0000", 2024-09-17T07:48:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4827000000000003e-05 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.4466954510000001, 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:11.782 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.440548583, 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:616 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'13, mlcod=49'11)", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.287418+0000", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.434364575, 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021074409999999998, 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.287418+0000", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.287418+0000", 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.287419+0000", 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.39e-07 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288253+0000", 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00083447600000000001 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288262+0000", 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7560000000000005e-06 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288266+0000", 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2590000000000002e-06 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288287+0000", 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1095999999999999e-05 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288298+0000", 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0492000000000001e-05 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.289490+0000", 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001192315 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.289526+0000", 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5608000000000002e-05 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.4282422280000002, 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.1530289649999999, 2024-09-17T07:48:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.147014848, 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.141378622, 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:730 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'24, mlcod=49'22)", 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.587515+0000", 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.1342675579999999, 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021140080000000001, 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.587515+0000", 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.587515+0000", 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.587516+0000", 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3e-07 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588304+0000", 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078832299999999995 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588315+0000", 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1054e-05 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588319+0000", 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5310000000000002e-06 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588364+0000", 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4926999999999999e-05 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588376+0000", 2024-09-17T07:48:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2245e-05 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.589602+0000", 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001226057 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.589629+0000", 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7240999999999999e-05 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:11.814 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:11.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:11.817 INFO:tasks.workunit.client.0.smithi098.stderr:+ used=67108864 2024-09-17T07:48:11.817 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-17T07:48:11.817 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 27914 2024-09-17T07:48:11.817 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:11.817 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=27914 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:11.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:11.819 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^27914\b' 2024-09-17T07:48:11.819 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^27914\b' 2024-09-17T07:48:11.819 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:11.815+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:11.840 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:11.840 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:11.840 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *27914\b' 2024-09-17T07:48:11.840 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *27914\b' 2024-09-17T07:48:11.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:11.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:11.916 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:11.916+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:11.920 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:11.921 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:11.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:12.017 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:12.016+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:12.070 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:12.070 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:12.070 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:12.070 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:12.070 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:12.078 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:12.119 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:12.119+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: marking collection for removal 2024-09-17T07:48:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:setting '_remove' omap key 2024-09-17T07:48:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:finish_remove_pgs 1.6_head removing 1.6 2024-09-17T07:48:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:Remove successful 2024-09-17T07:48:12.219 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:12.218+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:12.222 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:12.222 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:12.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:12.222 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:12.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.055708772999999, 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:12.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:12.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.048049257000001, 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:12.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.232 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.039659689, 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.039288439, 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:12.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.031225473999999, 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.030854361999999, 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.021573956999999, 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.019914524000001, 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.013889699, 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:12.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.013639372, 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.271 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.006525534, 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:12.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.006008435, 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:12.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.9870372679999999, 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:12.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:12.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.286 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.9807804079999993, 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:12.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.9742443460000008, 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.6013779360000004, 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:12.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.5994175290000001, 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.713748893, 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:724 3.2 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.567553+0000", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.6533648809999999, 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039753059999999996, 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 724 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.567553+0000", 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.567553+0000", 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.567554+0000", 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.09e-07 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.568390+0000", 2024-09-17T07:48:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083628999999999995 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.568402+0000", 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2244e-05 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.568409+0000", 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6159999999999998e-06 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.568450+0000", 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0669e-05 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.568522+0000", 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2540999999999995e-05 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.568608+0000", 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5599999999999994e-05 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.570724+0000", 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021164320000000001 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.571473+0000", 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074858599999999998 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.571505+0000", 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2063000000000001e-05 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.571528+0000", 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3356000000000001e-05 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:730 3.2 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head [write 3276800~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.585867+0000", 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.6350502790000001, 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039384809999999998, 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 730 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.585867+0000", 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.585867+0000", 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.585868+0000", 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4200000000000004e-07 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.587046+0000", 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001177483 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.587057+0000", 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.092e-05 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.587063+0000", 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:48:12.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.587105+0000", 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2707000000000003e-05 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.587229+0000", 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012336899999999999 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.587355+0000", 2024-09-17T07:48:12.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000126453 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.589399+0000", 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020438370000000002 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.589732+0000", 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033290400000000002 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.589767+0000", 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5692999999999999e-05 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.589806+0000", 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8417000000000002e-05 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.326 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:12.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:12.328 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:12.328+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:12.352 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:12.353 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:12.353 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:12.353 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:12.354 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:12.356 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:12.356 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:12.356 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:12.428 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:12.428+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:12.471 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:12.471 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:12.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:12.529 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:12.529+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:12.580 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:12.581 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:12.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:12.631 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:12.630+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:12.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-17T07:48:12.649 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:12.649 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:12.649 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:12.650 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:12.650 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:12.650 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:12.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28076 2024-09-17T07:48:12.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28076 2024-09-17T07:48:12.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28076 -C rbd-nbd 2024-09-17T07:48:12.712 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:12.712 INFO:tasks.workunit.client.0.smithi098.stdout: 28076 ? 00:00:00 rbd-nbd 2024-09-17T07:48:12.713 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:12.713 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-17T07:48:12.731 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:12.731+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:12.734 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.475848733999999, 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.1075431179999997, 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.4153608630000001, 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:12.750 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.4095040069999998, 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:632 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.329360+0000", 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.4037270590000004, 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034240569999999999, 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 632 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.329360+0000", 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.329360+0000", 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.329360+0000", 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.330223+0000", 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086266800000000003 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.330233+0000", 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7380000000000006e-06 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.330237+0000", 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9179999999999996e-06 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.330260+0000", 2024-09-17T07:48:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3774999999999999e-05 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.330315+0000", 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4666000000000001e-05 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.330388+0000", 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2801999999999999e-05 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.331738+0000", 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001350003 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.332717+0000", 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097934099999999989 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.332743+0000", 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6284000000000001e-05 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.332784+0000", 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0207999999999999e-05 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3978110480000003, 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:12.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:12.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:656 3.0 e49/16 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head v 49'42, mlcod=49'40)", 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.393376+0000", 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3397101979999997, 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023057149999999998, 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.769 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393376+0000", 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393376+0000", 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393377+0000", 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9100000000000003e-07 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393932+0000", 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055450200000000001 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393947+0000", 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4915e-05 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393954+0000", 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1919999999999999e-06 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.393998+0000", 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4527000000000002e-05 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.394021+0000", 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2872000000000001e-05 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.395652+0000", 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001630802 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.395682+0000", 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0213999999999999e-05 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:658 3.0 e49/16 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head v 49'44, mlcod=49'42)", 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.399198+0000", 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3338886299999997, 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022402329999999999, 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399198+0000", 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399198+0000", 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399199+0000", 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9299999999999997e-07 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399391+0000", 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019227299999999999 2024-09-17T07:48:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399403+0000", 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323e-05 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399409+0000", 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8969999999999996e-06 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399451+0000", 2024-09-17T07:48:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.159e-05 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.399476+0000", 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5029e-05 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.401406+0000", 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019297439999999999 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.401438+0000", 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2684000000000001e-05 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3289564650000001, 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:662 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.410004+0000", 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3230822150000003, 2024-09-17T07:48:12.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003460355, 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 662 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410004+0000", 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410004+0000", 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410005+0000", 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410832+0000", 2024-09-17T07:48:12.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082713199999999995 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410843+0000", 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-05 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410847+0000", 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410880+0000", 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2515999999999998e-05 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.410945+0000", 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5248999999999998e-05 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.411016+0000", 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1444000000000002e-05 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.412505+0000", 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014887539999999999 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.413380+0000", 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00087454099999999995 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.413414+0000", 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.413465+0000", 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0782000000000003e-05 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3170571759999996, 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:12.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:12.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:12.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:666 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.421622+0000", 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.3114647310000001, 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034688150000000001, 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 666 2024-09-17T07:48:12.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.421622+0000", 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.421622+0000", 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.421623+0000", 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422447+0000", 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082444099999999998 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422457+0000", 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7259999999999997e-06 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422463+0000", 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3799999999999999e-06 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422498+0000", 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4754e-05 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422555+0000", 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6847e-05 2024-09-17T07:48:12.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422625+0000", 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9747999999999997e-05 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.424050+0000", 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001425812 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.425016+0000", 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000965294 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.425045+0000", 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9252999999999999e-05 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.425091+0000", 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5890999999999998e-05 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:676 3.0 e49/16 3:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head v 49'46, mlcod=49'44)", 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.445804+0000", 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.2872830969999995, 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022021200000000001, 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.803 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.445804+0000", 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.445804+0000", 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.445804+0000", 2024-09-17T07:48:12.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.44e-07 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.446010+0000", 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00020591399999999999 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.446022+0000", 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1891e-05 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.446026+0000", 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1110000000000001e-06 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.446067+0000", 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1084000000000002e-05 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.446090+0000", 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2694e-05 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.447972+0000", 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001881876 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.448006+0000", 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3905999999999997e-05 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.807 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:678 3.0 e49/16 3:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head v 49'48, mlcod=49'46)", 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.451473+0000", 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.2816137100000002, 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021557070000000002, 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451473+0000", 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451473+0000", 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451474+0000", 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1500000000000004e-07 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451690+0000", 2024-09-17T07:48:12.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00021676700000000001 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451705+0000", 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4769e-05 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451714+0000", 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2789999999999998e-06 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451749+0000", 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5871999999999998e-05 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.451769+0000", 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.914e-05 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.453583+0000", 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018148229999999999 2024-09-17T07:48:12.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.453629+0000", 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5342000000000001e-05 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:12.812 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.266212629, 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:12.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.818 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.2602044220000002, 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:12.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:12.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.2545040649999999, 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.825 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:12.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:12.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:12.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:12.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:12.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.2487644400000004, 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:12.832 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:12.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:12.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:700 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'55, mlcod=49'52)", 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.508714+0000", 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.2243728540000003, 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002226712, 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508714+0000", 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508714+0000", 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508714+0000", 2024-09-17T07:48:12.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4700000000000001e-07 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.509040+0000", 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032530999999999999 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.509052+0000", 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2021000000000001e-05 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.509058+0000", 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3280000000000002e-06 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.509103+0000", 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5207000000000003e-05 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.509123+0000", 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9850000000000001e-05 2024-09-17T07:48:12.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.510903+0000", 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017803140000000001 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.510941+0000", 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7135000000000003e-05 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4950.0:704 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'59, mlcod=49'57)", 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.520349+0000", 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.212737401, 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023265349999999998, 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520349+0000", 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520349+0000", 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520350+0000", 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520600+0000", 2024-09-17T07:48:12.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024992999999999999 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520615+0000", 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4538e-05 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520624+0000", 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5540000000000003e-06 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520672+0000", 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8199999999999999e-05 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.520695+0000", 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3093999999999999e-05 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.522642+0000", 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019468669999999999 2024-09-17T07:48:12.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.522676+0000", 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3596000000000002e-05 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:12.847 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:12.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:12.850 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:12.848+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ provisioned=67108864 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ used=67108864 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:12.851 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:12.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:12.852 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:48:12.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:48:12.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:12.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-17T07:48:12.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-09-17T07:48:12.891 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:12.891 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-17T07:48:12.933 INFO:tasks.workunit.client.0.smithi098.stderr:+ provisioned=67108864 2024-09-17T07:48:12.934 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd -p rbd --format xml du testrbdnbd27356 2024-09-17T07:48:12.934 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-17T07:48:12.949 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:12.949+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:12.951 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:12.951 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:12.951 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:12.951 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:12.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ used=0 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -lt 67108864 ']' 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 28076 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28076 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:12.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:12.976 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:12.977 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:12.977 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:12.977 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:13.050 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:13.050+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:13.056 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:13.056 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28076\b' 2024-09-17T07:48:13.056 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28076\b' 2024-09-17T07:48:13.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:13.094 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:13.095 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:13.095 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28076\b' 2024-09-17T07:48:13.095 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28076\b' 2024-09-17T07:48:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:13.105 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:13.105 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:13.111 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:13.152 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:13.152+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:13.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:48:13.160+0000 7f4d3a00e200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-17T07:48:13.169 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:13.185 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:13.185 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:13.201 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:13.201 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:13.201 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:13.217 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:13.233 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:13.236 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:13.251 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:13.251 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:13.252 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:13.251+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:13.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:13.353 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:13.353+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:13.398 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:13.399 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:13.407 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:13.453 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:13.453+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:13.505 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:13.505 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:13.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:13.555 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:13.555+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:13.606 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd27356 2024-09-17T07:48:13.607 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:13.607 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:13.607 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:13.608 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:13.609 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:13.609 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:13.609 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd27356 2024-09-17T07:48:13.652 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:13.652 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:13.655 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:13.655+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:13.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:13.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:48:13.711 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:13.712 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:13.712 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:13.712 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:13.712 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:13.712 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:13.756 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:13.756+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:13.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28238 2024-09-17T07:48:13.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28238 2024-09-17T07:48:13.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28238 -C rbd-nbd 2024-09-17T07:48:13.772 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:13.772 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:13.774 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:13.775 INFO:tasks.workunit.client.0.smithi098.stdout: 28238 ? 00:00:00 rbd-nbd 2024-09-17T07:48:13.775 INFO:tasks.workunit.client.0.smithi098.stderr:++ basename /dev/nbd0 2024-09-17T07:48:13.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ devname=nbd0 2024-09-17T07:48:13.776 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-17T07:48:13.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ blocks=65536 2024-09-17T07:48:13.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 65536 2024-09-17T07:48:13.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd resize rbd/testrbdnbd27356 --size 128M 2024-09-17T07:48:13.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:13.819 INFO:tasks.workunit.client.0.smithi098.stderr: Resizing image: 100% complete...done. 2024-09-17T07:48:13.826 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info rbd/testrbdnbd27356 2024-09-17T07:48:13.857 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:13.857+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:13.860 INFO:tasks.workunit.client.0.smithi098.stdout:rbd image 'testrbdnbd27356': 2024-09-17T07:48:13.860 INFO:tasks.workunit.client.0.smithi098.stdout: size 128 MiB in 32 objects 2024-09-17T07:48:13.860 INFO:tasks.workunit.client.0.smithi098.stdout: order 22 (4 MiB objects) 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: snapshot_count: 0 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: id: 123eb9c2b614 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: block_name_prefix: rbd_data.123eb9c2b614 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: format: 2 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: op_features: 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: flags: 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: create_timestamp: Tue Sep 17 07:48:01 2024 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: access_timestamp: Tue Sep 17 07:48:01 2024 2024-09-17T07:48:13.861 INFO:tasks.workunit.client.0.smithi098.stdout: modify_timestamp: Tue Sep 17 07:48:01 2024 2024-09-17T07:48:13.863 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-17T07:48:13.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ blocks2=131072 2024-09-17T07:48:13.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 131072 2024-09-17T07:48:13.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 131072 -eq 131072 2024-09-17T07:48:13.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd resize rbd/testrbdnbd27356 --allow-shrink --size 64M 2024-09-17T07:48:13.888 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:13.888 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:13.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:13.908 INFO:tasks.workunit.client.0.smithi098.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-09-17T07:48:13.920 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-17T07:48:13.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ blocks2=65536 2024-09-17T07:48:13.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 65536 2024-09-17T07:48:13.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 65536 -eq 65536 2024-09-17T07:48:13.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 28238 2024-09-17T07:48:13.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:13.923 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28238 2024-09-17T07:48:13.923 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:13.923 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:13.923 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:13.923 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:13.923 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:13.925 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:13.925 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:13.925 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:13.959 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:13.959+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:13.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:13.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28238\b' 2024-09-17T07:48:13.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28238\b' 2024-09-17T07:48:14.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:14.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:14.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28238\b' 2024-09-17T07:48:14.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28238\b' 2024-09-17T07:48:14.040 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:14.040 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:14.040 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.040 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:14.059 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:14.059+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:14.150 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.150 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.158 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:14.161 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:14.160+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:14.259 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.259 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.262 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:14.262+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:14.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:14.363 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:14.362+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:14.368 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.368 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:14.465 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:14.464+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:14.475 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.475 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:14.541 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd27356 2024-09-17T07:48:14.541 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:14.542 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:14.542 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:14.543 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:14.544 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:14.545 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:14.545 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd27356 2024-09-17T07:48:14.566 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:14.566+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:14.622 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.622 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:14.638 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:14.639 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd list 2024-09-17T07:48:14.639 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk -v pool=rbd -v img=testrbdnbd27356 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-09-17T07:48:14.662 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:48:14.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:48:14.668+0000 7f4d3a00e200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-17T07:48:14.669 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:14.669+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:14.688 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28368 2024-09-17T07:48:14.689 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28368 2024-09-17T07:48:14.689 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28368 -C rbd-nbd 2024-09-17T07:48:14.699 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:14.699 INFO:tasks.workunit.client.0.smithi098.stdout: 28368 ? 00:00:00 rbd-nbd 2024-09-17T07:48:14.699 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-17T07:48:14.699 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:14.699 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:14.700 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:14.700 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:48:14.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:48:14.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:14.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-17T07:48:14.767 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:14.766+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:14.773 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.773 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:14.787 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:14.787 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:14.787 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.0758527 s, 885 MB/s 2024-09-17T07:48:14.788 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:14.788 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:14.789 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:14.789 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:14.789 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:14.790 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:48:14.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:48:14.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:14.792 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:14.801 INFO:tasks.workunit.client.0.smithi098.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-09-17T07:48:14.801 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:48:14.801 INFO:tasks.workunit.client.0.smithi098.stderr:0+0 records out 2024-09-17T07:48:14.801 INFO:tasks.workunit.client.0.smithi098.stderr:0 bytes copied, 0.000451315 s, 0.0 kB/s 2024-09-17T07:48:14.802 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:14.802 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 28368 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28368 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:14.803 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:14.805 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:14.805 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:14.805 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:14.869 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:14.867+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:14.869 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:14.869 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28368\b' 2024-09-17T07:48:14.869 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28368\b' 2024-09-17T07:48:14.882 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.882 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.891 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:14.907 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:14.908 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:14.908 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28368\b' 2024-09-17T07:48:14.908 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28368\b' 2024-09-17T07:48:14.918 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:14.918 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:14.969 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:14.968+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:14.989 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:14.989 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:14.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:15.070 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:15.069+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:15.095 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:15.095 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:15.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:15.170 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:15.170+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:15.200 INFO:teuthology.orchestra.run.smithi094.stdout:2.0 2024-09-17T07:48:15.201 INFO:teuthology.orchestra.run.smithi094.stdout:1.0 2024-09-17T07:48:15.208 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:15.208 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:15.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:15.237 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:15.303 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:15.302+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:15.319 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:15.319 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:15.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:15.376 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:15.377 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:15.375+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:15.393 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:15.393 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:15.393 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:15.409 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:15.409 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:15.419 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd27356 2024-09-17T07:48:15.419 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:15.420 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:15.421 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:15.421 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:15.423 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:15.423 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:15.423 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd27356 2024-09-17T07:48:15.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:15.425 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:15.428 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.298221063, 2024-09-17T07:48:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.290561546999999, 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.282171978999999, 2024-09-17T07:48:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.281800729, 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.273737764, 2024-09-17T07:48:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.273366652, 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.264086247, 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.262426813999999, 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.256401989, 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.256151662000001, 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.249037824, 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.248520725000001, 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.229549558, 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.223292698, 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.216756636, 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.8438902260000001, 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.8419298189999997, 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 6.9562611829999996, 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.6454155450000001, 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.6440541, 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.574 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:15.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:15.577 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:15.575+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:15.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:15.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:15.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:15.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:15.578 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:15.578 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:15.578 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28448 2024-09-17T07:48:15.578 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28448 2024-09-17T07:48:15.578 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28448 -C rbd-nbd 2024-09-17T07:48:15.584 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:15.584 INFO:tasks.workunit.client.0.smithi098.stdout: 28448 ? 00:00:00 rbd-nbd 2024-09-17T07:48:15.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:15.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:15.585 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:15.585 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:15.585 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:48:15.587 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:48:15.587 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:15.587 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-17T07:48:15.676 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:15.676+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:15.722 INFO:tasks.thrashosds.thrasher:Moving pg 1.6 from osd.0 to osd.3 2024-09-17T07:48:15.722 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi026.front.sepia.ceph.com to ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:48:15.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4886.0:90 3.0 e53/16 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head v 53'106, mlcod=53'104)", 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.729136+0000", 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.001096315, 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011140220000000001, 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729136+0000", 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729136+0000", 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729137+0000", 2024-09-17T07:48:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5799999999999999e-07 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729376+0000", 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00023902 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729392+0000", 2024-09-17T07:48:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6419999999999999e-05 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729399+0000", 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8859999999999998e-06 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729446+0000", 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6251999999999997e-05 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.729470+0000", 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4119999999999999e-05 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-09-17T07:48:15.734 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:15.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:15.752 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-09-17T07:48:15.787 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-17T07:48:15.787 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:15.786+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:15.811 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.ZkKvHm66bF 2024-09-17T07:48:15.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/log/ceph/exp.1.6.0 /tmp/tmp.ZkKvHm66bF 2024-09-17T07:48:15.878 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:15.877+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:15.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.ZkKvHm66bF 2024-09-17T07:48:15.904 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:15.905 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:15.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:15.943 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records in 2024-09-17T07:48:15.944 INFO:tasks.workunit.client.0.smithi098.stderr:64+0 records out 2024-09-17T07:48:15.944 INFO:tasks.workunit.client.0.smithi098.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.347074 s, 193 MB/s 2024-09-17T07:48:15.945 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false timeout 10 rbd bench testrbdnbd27356 --io-type write --io-size=1024 --io-total=1024 2024-09-17T07:48:15.945 INFO:tasks.workunit.client.0.smithi098.stderr:+ timeout 10 rbd bench testrbdnbd27356 --io-type write --io-size=1024 --io-total=1024 2024-09-17T07:48:15.989 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:48:15.988+0000 7f30ba94a640 -1 librbd::exclusive_lock::ImageDispatch: 0x7f30940020d0 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-09-17T07:48:15.989 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: failed to flush: 2024-09-17T07:48:15.988+0000 7f30ba94a640 -1 librbd::io::AioCompletion: 0x562d43f7b470 fail: (30) Read-only file system 2024-09-17T07:48:15.989 INFO:tasks.workunit.client.0.smithi098.stderr:(30) Read-only file system 2024-09-17T07:48:15.989 INFO:tasks.workunit.client.0.smithi098.stderr:bench failed: (30) Read-only file system 2024-09-17T07:48:15.993 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:15.993 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 28448 2024-09-17T07:48:15.993 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:48:15.994 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28448 2024-09-17T07:48:15.994 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:15.994 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:15.994 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:15.994 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:15.994 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:15.995 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:15.996 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:15.996 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:48:16.017 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:16.015+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:16.019 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.ZkKvHm66bF is 6KB 2024-09-17T07:48:16.065 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:16.065 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28448\b' 2024-09-17T07:48:16.065 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28448\b' 2024-09-17T07:48:16.069 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.ZkKvHm66bF 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:16.078 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.819002343999999, 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:16.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:16.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4506967280000005, 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:16.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:16.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:16.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:16.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.758514473, 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:16.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.097 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.7526576169999997, 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:16.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:16.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.7409646580000002, 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:16.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.109 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.672110075, 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:16.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:16.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:16.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:16.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.115 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.6602107860000004, 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:16.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:666 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.121 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.421622+0000", 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.6546183409999999, 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034688150000000001, 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 666 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.421622+0000", 2024-09-17T07:48:16.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.421622+0000", 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.421623+0000", 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422447+0000", 2024-09-17T07:48:16.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082444099999999998 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422457+0000", 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7259999999999997e-06 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422463+0000", 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3799999999999999e-06 2024-09-17T07:48:16.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422498+0000", 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4754e-05 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422555+0000", 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6847e-05 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.422625+0000", 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9747999999999997e-05 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.424050+0000", 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001425812 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.425016+0000", 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000965294 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.425045+0000", 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9252999999999999e-05 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.425091+0000", 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5890999999999998e-05 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.127 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.6093662389999999, 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:16.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:16.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:16.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.133 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.603358032, 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.5976576749999998, 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.139 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:16.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:16.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 7.5919180500000003, 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:16.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:16.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.2588070020000002, 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:16.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.156 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.41370610499999999, 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.40761938199999997, 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:16.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.32416308100000002, 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:16.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:16.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:16.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.31784864899999998, 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:16.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:16.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:16.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.31147718899999999, 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.30555471299999998, 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.247571817, 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:16.199 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:16.201 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:16.202 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:16.202 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28448\b' 2024-09-17T07:48:16.202 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28448\b' 2024-09-17T07:48:16.202 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:16.202 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:16.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:16.204 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:16.200+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:16.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.6.0 2024-09-17T07:48:16.301 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:16.301+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:16.306 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:16.306 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:16.306 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:16.306 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:16.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:16.403 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:16.402+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:16.453 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:16.453 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:16.454 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:16.454 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:16.454 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:16.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:16.504 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:16.503+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:16.604 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:16.604+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:16.615 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:16.615 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd bench testrbdnbd27356 --io-type write --io-size=1024 --io-total=1024 2024-09-17T07:48:16.618 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:16.618 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:16.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.451727334999999, 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.619 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:16.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.444388001, 2024-09-17T07:48:16.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:16.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:16.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:16.627 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.443670926999999, 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:16.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:16.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.434443366, 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:16.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.425218798, 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:16.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.641 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.409744158000001, 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:16.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:16.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.400454648, 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.393864403, 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:16.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:16.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:16.655 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.377681922000001, 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:16.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.368607001999999, 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:16.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0281681260000006, 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:16.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:16.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:16.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.026115849, 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9658841979999995, 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:16.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:16.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.8047050489999998, 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:16.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.804680748, 2024-09-17T07:48:16.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:16.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.8046463720000001, 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:16.701 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.8046250150000001, 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.702 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:16.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.8019563710000002, 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:16.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:93 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.712 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.799455531, 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012110549, 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8100000000000002e-07 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817503+0000", 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3290000000000004e-06 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817513+0000", 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0672999999999999e-05 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817517+0000", 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7900000000000001e-06 2024-09-17T07:48:16.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.825713+0000", 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081956609999999999 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.825785+0000", 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1815000000000003e-05 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.829543+0000", 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037588460000000001 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.829573+0000", 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9302000000000002e-05 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.829608+0000", 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5451999999999999e-05 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7922387980000001, 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:16.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:16.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:16.723 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:16.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:48:16.725 INFO:tasks.workunit.client.0.smithi098.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-17T07:48:16.725 INFO:tasks.workunit.client.0.smithi098.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-17T07:48:16.726 INFO:tasks.workunit.client.0.smithi098.stdout:elapsed: 0 ops: 1 ops/sec: 999.995 bytes/sec: 1000 KiB/s 2024-09-17T07:48:16.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:16.727 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:16.725+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:16.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:16.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:16.765 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:16.765 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:16.765 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:16.765 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:16.765 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:16.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28566 2024-09-17T07:48:16.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28566 2024-09-17T07:48:16.818 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28566 -C rbd-nbd 2024-09-17T07:48:16.826 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:16.826+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:16.828 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:16.828 INFO:tasks.workunit.client.0.smithi098.stdout: 28566 ? 00:00:00 rbd-nbd 2024-09-17T07:48:16.828 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device testrbdnbd27356 28566 2024-09-17T07:48:16.829 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=testrbdnbd27356 2024-09-17T07:48:16.829 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28566 2024-09-17T07:48:16.829 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27356 2024-09-17T07:48:16.829 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:16.829 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:16.829 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:16.830 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:16.831 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:16.831 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:16.831 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27356 2024-09-17T07:48:16.865 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:16.865 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:16.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:16.904 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:16.905 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28566\b' 2024-09-17T07:48:16.905 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28566\b' 2024-09-17T07:48:16.942 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:16.943 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:16.943 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28566\b' 2024-09-17T07:48:16.943 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28566\b' 2024-09-17T07:48:16.953 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:16.953 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:16.957 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:16.957+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:17.011 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:17.011 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:17.011 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:17.011 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:17.011 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:17.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:17.060 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:17.060+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:17.129 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:17.129+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.977337202999999, 2024-09-17T07:48:17.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.969197165000001, 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:17.169 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.949574473, 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:17.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.942249241000001, 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.935637140000001, 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.931455817, 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.929143173, 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:17.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.904096069, 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:17.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:17.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.903758612000001, 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:17.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.5702151840000003, 2024-09-17T07:48:17.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:17.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:17.206 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.4761831880000003, 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.8839525419999994, 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:17.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:17.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.8778056739999993, 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:17.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:616 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'13, mlcod=49'11)", 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.287418+0000", 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.8716216659999994, 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021074409999999998, 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.287418+0000", 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.287418+0000", 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.287419+0000", 2024-09-17T07:48:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.39e-07 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288253+0000", 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00083447600000000001 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288262+0000", 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7560000000000005e-06 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288266+0000", 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2590000000000002e-06 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288287+0000", 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1095999999999999e-05 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.288298+0000", 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0492000000000001e-05 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.289490+0000", 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001192315 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.289526+0000", 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5608000000000002e-05 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.8654993189999995, 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.5902860560000001, 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.5842719390000006, 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.5786357130000006, 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:730 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'24, mlcod=49'22)", 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.587515+0000", 2024-09-17T07:48:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.5715246490000006, 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021140080000000001, 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.587515+0000", 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.587515+0000", 2024-09-17T07:48:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.587516+0000", 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3e-07 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588304+0000", 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078832299999999995 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588315+0000", 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1054e-05 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588319+0000", 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5310000000000002e-06 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588364+0000", 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4926999999999999e-05 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.588376+0000", 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2245e-05 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.589602+0000", 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001226057 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.589629+0000", 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7240999999999999e-05 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.537315835, 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:17.250 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:17.251 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:17.251+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:17.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:17.352 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:17.352+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:17.355 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:17.355 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:17.363 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:17.364 INFO:teuthology.orchestra.run.smithi094.stdout:get_pg_num_history pg_num_history pg_num_history(e49 pg_nums {1={12=8},2={12=1},3={16=4},4={46=8}} deleted_pools ) 2024-09-17T07:48:17.364 INFO:teuthology.orchestra.run.smithi094.stdout:Importing pgid 1.6 2024-09-17T07:48:17.364 INFO:teuthology.orchestra.run.smithi094.stdout:write_pg epoch 30 info 1.6( empty local-lis/les=29/30 n=0 ec=12/12 lis/c=29/29 les/c/f=30/30/0 sis=29) 2024-09-17T07:48:17.364 INFO:teuthology.orchestra.run.smithi094.stdout:Import successful 2024-09-17T07:48:17.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:17.455 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:17.455 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/testrbdnbd27356@snap 2024-09-17T07:48:17.466 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:17.466 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:17.467 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:17.467 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:17.467 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:17.467 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:17.465+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:17.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:17.533 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:17.549 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:17.554 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:17.552+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:17.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:17.566 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:17.567 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:17.582 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:17.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:17.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:17.598 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:17.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:17.620 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:17.620 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:17.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:17.655 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:17.655+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:17.670 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:48:17.681 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-17T07:48:17.688 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356@snap 2024-09-17T07:48:17.689 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:17.689 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:17.689 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:17.690 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:17.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:17.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:17.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356@snap 2024-09-17T07:48:17.756 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:17.756+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:17.774 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:17.774 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:17.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:17.785 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:17.786 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:17.786 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:17.786 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:17.786 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:17.786 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:17.836 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28667 2024-09-17T07:48:17.836 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28667 2024-09-17T07:48:17.836 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28667 -C rbd-nbd 2024-09-17T07:48:17.846 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:17.846 INFO:tasks.workunit.client.0.smithi098.stdout: 28667 ? 00:00:00 rbd-nbd 2024-09-17T07:48:17.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device testrbdnbd27356@snap 28667 2024-09-17T07:48:17.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=testrbdnbd27356@snap 2024-09-17T07:48:17.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28667 2024-09-17T07:48:17.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27356@snap 2024-09-17T07:48:17.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:17.847 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:17.848 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:17.848 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:17.849 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:17.849 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:17.849 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27356@snap 2024-09-17T07:48:17.857 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:17.856+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:17.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /var/log/ceph/exp.1.6.0 2024-09-17T07:48:17.888 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:17.888 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:17.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:17.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /var/log/ceph/exp.1.6.0 2024-09-17T07:48:17.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:17.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28667\b' 2024-09-17T07:48:17.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28667\b' 2024-09-17T07:48:17.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:48:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28667\b' 2024-09-17T07:48:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28667\b' 2024-09-17T07:48:17.969 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:17.969 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:17.994 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:17.993+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:18.053 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.053 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.058 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:18.057+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:18.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:18.160 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:18.160+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:18.161 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.161 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:18.222 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:48:18.222 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":54,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:48:17.665258+0000","last_up_change":"2024-09-17T07:48:08.647128+0000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"new_default_pool27356","create_time":"2024-09-17T07:48:02.539579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":29,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":31884},{"type":"v1","addr":"172.21.15.26:6801","nonce":31884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":31884},{"type":"v1","addr":"172.21.15.26:6803","nonce":31884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":31884},{"type":"v1","addr":"172.21.15.26:6807","nonce":31884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":31884},{"type":"v1","addr":"172.21.15.26:6805","nonce":31884}]},"public_addr":"172.21.15.26:6801/31884","cluster_addr":"172.21.15.26:6803/31884","heartbeat_back_addr":"172.21.15.26:6807/31884","heartbeat_front_addr":"172.21.15.26:6805/31884","state":["exists"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":0,"in":1,"weight":0.3944549560546875,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":31604},{"type":"v1","addr":"172.21.15.94:6801","nonce":31604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":31604},{"type":"v1","addr":"172.21.15.94:6803","nonce":31604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":31604},{"type":"v1","addr":"172.21.15.94:6807","nonce":31604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":31604},{"type":"v1","addr":"172.21.15.94:6805","nonce":31604}]},"public_addr":"172.21.15.94:6801/31604","cluster_addr":"172.21.15.94:6803/31604","heartbeat_back_addr":"172.21.15.94:6807/31604","heartbeat_front_addr":"172.21.15.94:6805/31604","state":["exists"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":0.456390380859375,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":31606},{"type":"v1","addr":"172.21.15.94:6817","nonce":31606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":31606},{"type":"v1","addr":"172.21.15.94:6819","nonce":31606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":31606},{"type":"v1","addr":"172.21.15.94:6823","nonce":31606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":31606},{"type":"v1","addr":"172.21.15.94:6821","nonce":31606}]},"public_addr":"172.21.15.94:6817/31606","cluster_addr":"172.21.15.94:6819/31606","heartbeat_back_addr":"172.21.15.94:6823/31606","heartbeat_front_addr":"172.21.15.94:6821/31606","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":50},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.5","osds":[1,0]}],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.2","osds":[2,4]},{"pgid":"3.1","osds":[2,4]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:48:18.234 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-17T07:48:18.235 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool rbd 2024-09-17T07:48:18.261 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:18.259+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:18.311 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.311 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.320 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:18.361 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:18.360+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:18.416 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.416 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:18.463 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:18.462+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:18.470 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:18.471 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd snap ls rbd/testrbdnbd27356 2024-09-17T07:48:18.471 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-17T07:48:18.510 INFO:tasks.workunit.client.0.smithi098.stderr:+ SNAPID=4 2024-09-17T07:48:18.510 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd27356 2024-09-17T07:48:18.510 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:18.510 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:18.511 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:18.511 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:18.513 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:18.513 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:18.513 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd27356 2024-09-17T07:48:18.523 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.523 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:18.564 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:18.563+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:18.608 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:18.608 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:18.608 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:18.608 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:18.608 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:18.608 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:18.634 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.634 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:18.661 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28770 2024-09-17T07:48:18.661 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28770 2024-09-17T07:48:18.661 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28770 -C rbd-nbd 2024-09-17T07:48:18.671 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:18.672 INFO:tasks.workunit.client.0.smithi098.stdout: 28770 ? 00:00:00 rbd-nbd 2024-09-17T07:48:18.672 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device '--snap-id 4 testrbdnbd27356' 28770 2024-09-17T07:48:18.672 INFO:tasks.workunit.client.0.smithi098.stderr:+ local 'args=--snap-id 4 testrbdnbd27356' 2024-09-17T07:48:18.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28770 2024-09-17T07:48:18.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 testrbdnbd27356 2024-09-17T07:48:18.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:18.673 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:18.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:18.673 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:18.675 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:18.675 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:18.675 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 testrbdnbd27356 2024-09-17T07:48:18.689 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:18.689+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:18.750 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.750 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:18.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28770\b' 2024-09-17T07:48:18.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28770\b' 2024-09-17T07:48:18.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:18.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:18.792 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:18.792 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28770\b' 2024-09-17T07:48:18.792 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28770\b' 2024-09-17T07:48:18.800 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:18.799+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:18.802 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:18.802 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:18.859 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:18.859 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:18.866 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:18.865+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:18.867 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:18.967 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:18.965+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:19.009 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:19.009 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:19.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:19.068 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:19.068+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:19.159 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:19.159 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:19.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:19.169 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:19.169+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:19.269 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:19.269 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:19.270 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:19.268+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:19.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:19.303 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:19.304 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/ns/testrbdnbd27356@snap 2024-09-17T07:48:19.335 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:48:19.371 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:19.371+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:19.420 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:19.421 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:19.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:19.472 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:19.472+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:19.530 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:19.530 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:19.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:19.583 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:19.583+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:19.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:19.642 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:19.642 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:19.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:19.680 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-17T07:48:19.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:48:19.687 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd27356@snap 2024-09-17T07:48:19.688 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:19.688 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:19.689 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:19.689 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:19.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:19.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:19.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd27356@snap 2024-09-17T07:48:19.692 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:19.691+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:19.723 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:19.723 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:19.741 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:19.741 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:19.741 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:19.757 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:19.757 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:19.773 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:19.773 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:19.774 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:19.774+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:19.776 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:19.789 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:19.790 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd ns 2024-09-17T07:48:19.790 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:19.790 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns=ns 2024-09-17T07:48:19.790 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:19.790 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:19.792 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:19.792 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:19.792 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:19.792 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:19.792 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:19.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:19.843 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28871 2024-09-17T07:48:19.843 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28871 2024-09-17T07:48:19.843 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28871 -C rbd-nbd 2024-09-17T07:48:19.853 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:19.853 INFO:tasks.workunit.client.0.smithi098.stdout: 28871 ? 00:00:00 rbd-nbd 2024-09-17T07:48:19.854 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device rbd/ns/testrbdnbd27356@snap 28871 2024-09-17T07:48:19.854 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=rbd/ns/testrbdnbd27356@snap 2024-09-17T07:48:19.854 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28871 2024-09-17T07:48:19.854 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd27356@snap 2024-09-17T07:48:19.854 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:19.854 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:19.855 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:19.855 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:19.856 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool .mgr 2024-09-17T07:48:19.857 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:19.857 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:19.857 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd27356@snap 2024-09-17T07:48:19.875 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:19.873+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:19.924 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:19.924 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28871\b' 2024-09-17T07:48:19.924 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28871\b' 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.779368527999999, 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.950 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.772029194000002, 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:19.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.771312120000001, 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.955 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:19.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.762084559000002, 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.964 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.752859991000001, 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:19.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.737385351, 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.969 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.728095840999998, 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:19.973 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:19.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.977 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.721505596, 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:19.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:19.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.705323114999999, 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:19.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:19.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.696248194999999, 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.986 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:19.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.355809319, 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:19.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:19.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:19.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:19.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:19.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.353757042, 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:19.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:19.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:19.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:19.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.293525390999999, 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:20.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:20.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:20.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.1323462419999997, 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.1323219409999998, 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.013 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:20.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:20.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.1322875650000004, 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.018 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:20.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:20.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.1322662079999999, 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:20.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.129597564, 2024-09-17T07:48:20.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:93 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.1270967240000003, 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012110549, 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 93 2024-09-17T07:48:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817498+0000", 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8100000000000002e-07 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817503+0000", 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3290000000000004e-06 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817513+0000", 2024-09-17T07:48:20.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0672999999999999e-05 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.817517+0000", 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7900000000000001e-06 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.825713+0000", 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081956609999999999 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.825785+0000", 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1815000000000003e-05 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.829543+0000", 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037588460000000001 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.829573+0000", 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9302000000000002e-05 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.829608+0000", 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5451999999999999e-05 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:20.039 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.1198799910000004, 2024-09-17T07:48:20.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:20.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:20.047 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:20.047 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:20.047 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28871\b' 2024-09-17T07:48:20.048 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28871\b' 2024-09-17T07:48:20.048 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:20.048 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:20.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:20.048 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:20.046+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:20.147 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:20.146+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:20.186 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:20.187 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:20.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:20.187 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:20.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:20.248 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:20.248+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:20.301 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:20.301 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:20.301 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:20.301 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:20.301 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:20.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:20.349 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:20.348+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:20.449 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.266750153, 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.258610115, 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.238987423000001, 2024-09-17T07:48:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.231662191000002, 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.22505009, 2024-09-17T07:48:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.220868766999999, 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.218556122999999, 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.193509019, 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.193171562, 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.859628133999999, 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.765596137999999, 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.173365492, 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.167218624, 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.154912269, 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 11.879699005999999, 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 11.873684889, 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 11.868048663, 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.8267287850000002, 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.66088312599999999, 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.65470101400000003, 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:20.549 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:20.550 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:20.549+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:20.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:20.552 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:20.552 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd snap ls rbd/ns/testrbdnbd27356 2024-09-17T07:48:20.552 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-17T07:48:20.552 INFO:tasks.workunit.client.0.smithi098.stderr:+ SNAPID=5 2024-09-17T07:48:20.552 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd27356 2024-09-17T07:48:20.552 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:20.553 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:20.553 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:20.553 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:20.553 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:20.553 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:20.553 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd27356 2024-09-17T07:48:20.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:20.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd ns 2024-09-17T07:48:20.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:20.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns=ns 2024-09-17T07:48:20.614 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:20.614 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:20.651 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:20.649+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:20.665 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:20.665 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:20.665 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:20.665 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:20.668 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=28959 2024-09-17T07:48:20.668 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 28959 2024-09-17T07:48:20.668 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 28959 -C rbd-nbd 2024-09-17T07:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:48:20.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:20.678 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:20.678 INFO:tasks.workunit.client.0.smithi098.stdout: 28959 ? 00:00:00 rbd-nbd 2024-09-17T07:48:20.679 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device '--snap-id 5 rbd/ns/testrbdnbd27356' 28959 2024-09-17T07:48:20.679 INFO:tasks.workunit.client.0.smithi098.stderr:+ local 'args=--snap-id 5 rbd/ns/testrbdnbd27356' 2024-09-17T07:48:20.679 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=28959 2024-09-17T07:48:20.680 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd27356 2024-09-17T07:48:20.680 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:20.680 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:20.680 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:20.680 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:20.682 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:20.682 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:20.682 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd27356 2024-09-17T07:48:20.754 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:20.754 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^28959\b' 2024-09-17T07:48:20.754 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^28959\b' 2024-09-17T07:48:20.771 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:20.770+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:20.792 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:20.793 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:20.793 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *28959\b' 2024-09-17T07:48:20.793 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *28959\b' 2024-09-17T07:48:20.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:20.803 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:20.829 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:20.829 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:20.829 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:20.829 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:20.830 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:20.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:20.879 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:20.878+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:20.954 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:20.953+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.815085853999999, 2024-09-17T07:48:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:20.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:20.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:20.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.807426337999999, 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.988 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:20.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:20.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:20.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:20.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.799036770000001, 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:20.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.79866552, 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.790602555, 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.790231443, 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.780951038000001, 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.779291605000001, 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.77326678, 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.773016453, 2024-09-17T07:48:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.036 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.765902615000002, 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:21.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.765385515999998, 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.043 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:21.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.746414348999998, 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:21.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:21.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.740157489000001, 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:21.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:21.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:21.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:21.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:21.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:21.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:21.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:21.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:21.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.733621426999999, 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.360755017000001, 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.35879461, 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 12.473125974, 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.1622803360000002, 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:21.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.1609188909999997, 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.101 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:21.102 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:48:21.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:21.103 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:21.101+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:21.203 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:21.202+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:21.244 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:21.245 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:21.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:21.304 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:21.303+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:21.304 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:21.305 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27356 2024-09-17T07:48:21.305 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:21.305 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:21.306 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:21.306 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:21.308 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:21.308 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:21.308 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27356 2024-09-17T07:48:21.354 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:21.355 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:21.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:21.404 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:21.404+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:21.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:21.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd ns 2024-09-17T07:48:21.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:21.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns=ns 2024-09-17T07:48:21.408 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:21.408 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:21.461 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29025 2024-09-17T07:48:21.461 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29025 2024-09-17T07:48:21.461 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29025 -C rbd-nbd 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:21.468 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.209401517, 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:21.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:21.472 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.841095900999999, 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:21.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:21.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.148913646, 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:21.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.143056789999999, 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.484 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:21.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.131363831, 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.490 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:21.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.062509248, 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.496 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:21.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.050609959000001, 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.502 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:21.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 12.999765412, 2024-09-17T07:48:21.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:21.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 12.993757205, 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:21.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 12.988056847999999, 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:21.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:21.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:21.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:21.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 12.982317223000001, 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:21.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:21.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.6492061749999998, 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:21.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.8041052779999998, 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:21.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:21.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:21.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.7980185549999996, 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:21.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.7145622539999996, 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.550 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:21.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:21.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.555 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.7082478219999997, 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:21.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.561 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.7018763620000001, 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.6959538859999999, 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.6379709900000003, 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:21.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.6802663180000001, 2024-09-17T07:48:21.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:21.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:21.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:21.583 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:21.584 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:21.585 INFO:tasks.workunit.client.0.smithi098.stdout: 29025 ? 00:00:00 rbd-nbd 2024-09-17T07:48:21.586 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool cache 2024-09-17T07:48:21.586 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:21.584+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:21.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:21.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd27356' 29025 2024-09-17T07:48:21.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd27356' 2024-09-17T07:48:21.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29025 2024-09-17T07:48:21.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27356 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27356 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29025\b' 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29025\b' 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29025\b' 2024-09-17T07:48:21.589 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29025\b' 2024-09-17T07:48:21.590 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:21.591 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:21.686 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:21.684+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:21.728 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:21.729 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:21.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:21.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:21.786 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:21.786+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:21.835 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:21.835 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:21.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:21.879 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:21.887 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:21.886+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:21.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:21.895 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:21.911 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:21.911 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:21.911 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:21.926 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:21.926 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:21.942 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:21.942 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:21.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:21.988 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:21.988+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:22.001 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.001 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:22.092 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27356 --snap snap 2024-09-17T07:48:22.092 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:22.092 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:22.093 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:22.093 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:22.095 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:22.095 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:22.095 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27356 --snap snap 2024-09-17T07:48:22.096 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:22.095+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:22.150 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.150 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.158 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:22.194 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:22.194 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd ns 2024-09-17T07:48:22.194 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:22.195 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns=ns 2024-09-17T07:48:22.195 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:22.195 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:22.199 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:22.199+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:22.246 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29091 2024-09-17T07:48:22.246 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29091 2024-09-17T07:48:22.246 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29091 -C rbd-nbd 2024-09-17T07:48:22.256 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:22.256 INFO:tasks.workunit.client.0.smithi098.stdout: 29091 ? 00:00:00 rbd-nbd 2024-09-17T07:48:22.257 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd27356 --snap snap' 29091 2024-09-17T07:48:22.257 INFO:tasks.workunit.client.0.smithi098.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd27356 --snap snap' 2024-09-17T07:48:22.257 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29091 2024-09-17T07:48:22.257 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27356 --snap snap 2024-09-17T07:48:22.257 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:22.257 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:22.258 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:22.258 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:22.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:22.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:22.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27356 --snap snap 2024-09-17T07:48:22.291 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:22.290+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:22.294 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.294 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.303 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:22.333 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:22.333 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29091\b' 2024-09-17T07:48:22.334 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29091\b' 2024-09-17T07:48:22.371 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:22.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:22.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29091\b' 2024-09-17T07:48:22.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29091\b' 2024-09-17T07:48:22.382 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:22.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:22.392 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:22.391+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:22.441 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.442 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:22.493 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:22.493+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:22.556 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.556 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:22.608 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:22.607+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:22.666 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.666 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:22.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:48:22.695 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:22.694+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:22.720 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:48:22.796 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:22.796+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:22.818 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.818 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:22.884 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:22.884 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:22.884 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:22.884 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:22.885 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:22.885 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:22.887 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:22.887 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:22.887 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:22.896 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:22.896+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:22.931 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:22.931 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:22.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:22.979 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:22.979 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:22.979 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:22.979 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:22.979 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:22.979 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:22.997 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:22.996+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:23.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29157 2024-09-17T07:48:23.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29157 2024-09-17T07:48:23.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29157 -C rbd-nbd 2024-09-17T07:48:23.042 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:23.042 INFO:tasks.workunit.client.0.smithi098.stdout: 29157 ? 00:00:00 rbd-nbd 2024-09-17T07:48:23.043 INFO:tasks.workunit.client.0.smithi098.stderr:+ pid=29157 2024-09-17T07:48:23.043 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd27356 2024-09-17T07:48:23.043 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:23.043 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:23.044 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.044 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.044 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:23.044 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:23.046 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:23.046 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:23.046 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd27356 2024-09-17T07:48:23.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:23.098 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:23.098+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:23.147 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd1 2024-09-17T07:48:23.147 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd ns 2024-09-17T07:48:23.147 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:23.147 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns=ns 2024-09-17T07:48:23.147 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:23.147 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd1'\'']/id' 2024-09-17T07:48:23.158 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.158 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:23.199 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:23.198+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:23.209 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29205 2024-09-17T07:48:23.209 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29205 2024-09-17T07:48:23.209 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29205 -C rbd-nbd 2024-09-17T07:48:23.219 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:23.219 INFO:tasks.workunit.client.0.smithi098.stdout: 29157 ? 00:00:00 rbd-nbd 2024-09-17T07:48:23.219 INFO:tasks.workunit.client.0.smithi098.stdout: 29205 ? 00:00:00 rbd-nbd 2024-09-17T07:48:23.220 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device rbd/ns/testrbdnbd27356 29205 2024-09-17T07:48:23.220 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=rbd/ns/testrbdnbd27356 2024-09-17T07:48:23.220 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29205 2024-09-17T07:48:23.220 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd27356 2024-09-17T07:48:23.221 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:23.221 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:23.221 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:23.221 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:23.222 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:23.223 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:23.223 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd27356 2024-09-17T07:48:23.251 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool new_default_pool27356 2024-09-17T07:48:23.266 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.266 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.275 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:23.297 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:23.297 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29205\b' 2024-09-17T07:48:23.297 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29205\b' 2024-09-17T07:48:23.316 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:23.316+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:23.343 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:23.343 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:23.343 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29205\b' 2024-09-17T07:48:23.343 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29205\b' 2024-09-17T07:48:23.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:23.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:23.401 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:23.401+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:23.412 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.412 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:23.504 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:23.504+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:23.558 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.559 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:23.602 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:23.602+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:23.684 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:48:23.712 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.712 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:23.739 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:23.738+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:23.804 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:23.803+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:23.818 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.818 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:23.855 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:23.855 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device rbd/testrbdnbd27356 29157 2024-09-17T07:48:23.855 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=rbd/testrbdnbd27356 2024-09-17T07:48:23.855 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29157 2024-09-17T07:48:23.856 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd27356 2024-09-17T07:48:23.856 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:23.856 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:23.856 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:23.856 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:23.858 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:23.858 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:23.858 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd27356 2024-09-17T07:48:23.904 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:23.904+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:23.924 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:23.924 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:23.926 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:23.926 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29157\b' 2024-09-17T07:48:23.926 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29157\b' 2024-09-17T07:48:23.933 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:23.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:23.964 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:23.965 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:23.965 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29157\b' 2024-09-17T07:48:23.965 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29157\b' 2024-09-17T07:48:23.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:23.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:24.005 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:24.005+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:24.048 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:24.049 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:24.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:24.086 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:24.102 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:24.102 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:24.107 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:24.106+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:24.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:24.134 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:24.151 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:24.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:24.155 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:24.163 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:24.163 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:24.163 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:24.163 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:24.163 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:24.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:24.213 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:24.213+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:24.308 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:24.308+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.148341289000001, 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:24.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.319 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.140681773000001, 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.132292204999999, 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:24.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:24.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.131920954999998, 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:24.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.334 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.123857990000001, 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:24.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.123486878000001, 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.114206472999999, 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:24.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.112547039999999, 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:24.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:24.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:24.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:24.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.106522214999998, 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.353 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:24.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:24.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.106271887999998, 2024-09-17T07:48:24.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:24.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:24.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.09915805, 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.098640951, 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.079669784, 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.073412923999999, 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:24.382 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.066876862000001, 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:24.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.694010452000001, 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.388 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:24.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:24.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.692050044999998, 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:24.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:24.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 15.806381409, 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:24.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:24.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 11.495535771, 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 11.494174326, 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.417 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:48:24.419 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:24.419 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:24.418+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:24.476 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' rbd = rbd ']' 2024-09-17T07:48:24.476 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd27356 2024-09-17T07:48:24.477 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:24.477 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:24.477 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:24.478 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:24.479 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:24.479 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:24.480 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd27356 2024-09-17T07:48:24.519 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:24.518+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:24.563 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:24.563 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:24.563 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:24.563 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:24.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:24.574 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:24.575 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:24.575 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:24.575 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:24.575 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:24.575 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:24.620 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:24.619+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:24.626 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29305 2024-09-17T07:48:24.626 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29305 2024-09-17T07:48:24.626 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29305 -C rbd-nbd 2024-09-17T07:48:24.636 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:24.636 INFO:tasks.workunit.client.0.smithi098.stdout: 29305 ? 00:00:00 rbd-nbd 2024-09-17T07:48:24.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device testrbdnbd27356 29305 2024-09-17T07:48:24.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=testrbdnbd27356 2024-09-17T07:48:24.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29305 2024-09-17T07:48:24.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27356 2024-09-17T07:48:24.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:24.637 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:24.638 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:24.638 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:24.639 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:24.639 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:24.640 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27356 2024-09-17T07:48:24.674 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:24.674 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:24.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:24.714 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:24.714 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29305\b' 2024-09-17T07:48:24.714 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29305\b' 2024-09-17T07:48:24.725 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:24.724+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:24.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:24.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:24.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29305\b' 2024-09-17T07:48:24.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29305\b' 2024-09-17T07:48:24.763 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:24.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:24.822 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:24.820+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:24.826 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:24.826 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:24.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:24.826 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.567827135000002, 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:24.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.199521519000001, 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.507339263999999, 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.501482408000001, 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.489789449, 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.420934866, 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:24.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.409035577000001, 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.35819103, 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:24.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:24.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.352182823, 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.872 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:24.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:24.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:24.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.346482466000001, 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:24.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:24.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:24.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.340742841000001, 2024-09-17T07:48:24.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:24.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:24.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 12.007631793, 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:24.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.1625308959999998, 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:24.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:24.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.1564441730000006, 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:24.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.0729878720000006, 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:24.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.0666734400000006, 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.0603019800000002, 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.0543795039999999, 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:24.932 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.9963966079999995, 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:24.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:24.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:24.938 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.0386919360000002, 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:24.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:24.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:24.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:24.942 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:24.944 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:24.943+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:24.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:25.044 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:25.044+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:25.084 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:25.084 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:25.084 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:25.085 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:25.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:25.145 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:25.144+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:25.189 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:25.189 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:25.189 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:25.189 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:25.190 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:25.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:25.265 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:25.265 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd config global set global rbd_default_pool new_default_pool27356 2024-09-17T07:48:25.281 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:25.281+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:25.302 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd create --size 10M testrbdnbd27356 2024-09-17T07:48:25.339 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:25.339 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:25.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:25.339 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:25.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.172838215999999, 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:25.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:25.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.165498882000001, 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:25.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:25.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.164781808000001, 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.155554247000001, 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:25.353 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:25.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:25.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:25.357 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.146329679000001, 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.130855039, 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:25.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:25.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:25.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.121565529000002, 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:25.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:25.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.114975284, 2024-09-17T07:48:25.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:25.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.098792802999998, 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:25.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:25.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.089717882999999, 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.379 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:25.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.749279006999998, 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:25.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.747226730000001, 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.389 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:25.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:25.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.686995078999999, 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:25.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.52581593, 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:25.400 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:25.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:25.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:25.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.525791629, 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:25.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:25.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.525757253, 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:25.411 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.525735896, 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:25.420 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.523067252000001, 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:25.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.513349678999999, 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:25.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:25.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:25.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5472392900000003, 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:25.430 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.435 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:25.437 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:25.436+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:25.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:25.438 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd27356 2024-09-17T07:48:25.438 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:25.438 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:25.438 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:25.439 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:25.439 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:25.439 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:25.439 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd27356 2024-09-17T07:48:25.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:25.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid new_default_pool27356 2024-09-17T07:48:25.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=new_default_pool27356 2024-09-17T07:48:25.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:25.441 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:25.442 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool27356'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:25.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29435 2024-09-17T07:48:25.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29435 2024-09-17T07:48:25.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29435 -C rbd-nbd 2024-09-17T07:48:25.502 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:25.502 INFO:tasks.workunit.client.0.smithi098.stdout: 29435 ? 00:00:00 rbd-nbd 2024-09-17T07:48:25.503 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device testrbdnbd27356 29435 2024-09-17T07:48:25.503 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=testrbdnbd27356 2024-09-17T07:48:25.503 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29435 2024-09-17T07:48:25.503 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27356 2024-09-17T07:48:25.504 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:25.504 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:25.504 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:25.504 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:48:25.506 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:48:25.506 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:25.506 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27356 2024-09-17T07:48:25.537 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:25.537+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:25.577 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:25.577 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:25.578 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:25.578 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:25.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:25.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29435\b' 2024-09-17T07:48:25.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29435\b' 2024-09-17T07:48:25.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:25.618 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:25.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:48:25.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29435\b' 2024-09-17T07:48:25.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29435\b' 2024-09-17T07:48:25.629 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:25.629 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:48:25.638 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:25.638+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:25.699 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:48:25.724 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:25.724 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:25.724 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:25.724 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:25.724 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:25.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:25.743 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:25.743+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.649076791999999, 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.640936753999998, 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.621314062, 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.61398883, 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.607376728999999, 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.603195406000001, 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.600882762000001, 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.575835657999999, 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.575498200999998, 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.241954773, 2024-09-17T07:48:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.147922777000002, 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.555692131000001, 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.549545262999999, 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.537238907999999, 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.262025645000001, 2024-09-17T07:48:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.256011527999998, 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.250375301999998, 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.209055424000001, 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.0432097650000003, 2024-09-17T07:48:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.037027653, 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:25.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.924 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:25.924 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:25.924 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:25.925 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:25.925+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:25.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:26.026 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:26.025+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:26.028 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:26.028 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:26.028 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:26.028 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:26.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:26.127 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:26.127+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:26.130 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:48:26.130 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd config global rm global rbd_default_pool 2024-09-17T07:48:26.156 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:26.165 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:26.166 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:26.166 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:26.166 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:26.167 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:26.169 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:26.169 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:26.169 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:48:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:26.181 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:26.181 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:26.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:26.182 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:26.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:26.231 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:26.231+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:26.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:26.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:26.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:26.263 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:26.263 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:26.263 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:26.263 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:26.278 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:26.294 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:26.302 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:48:26.313 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:26.313 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:26.315 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29533 2024-09-17T07:48:26.315 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29533 2024-09-17T07:48:26.315 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29533 -C rbd-nbd 2024-09-17T07:48:26.325 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:26.325 INFO:tasks.workunit.client.0.smithi098.stdout: 29533 ? 00:00:00 rbd-nbd 2024-09-17T07:48:26.326 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo kill 29533 2024-09-17T07:48:26.326 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:26.326 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:26.327 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:26.327 INFO:tasks.workunit.client.0.smithi098.stderr:++ which kill 2024-09-17T07:48:26.329 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/kill 2024-09-17T07:48:26.329 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:26.329 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/kill 29533 2024-09-17T07:48:26.329 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:26.329 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:26.330 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:26.329+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:26.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:26.338 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:26.339 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:26.347 INFO:tasks.workunit.client.0.smithi098.stderr:++ seq 10 2024-09-17T07:48:26.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:26.352 INFO:tasks.workunit.client.0.smithi098.stderr:+ for i in `seq 10` 2024-09-17T07:48:26.352 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:26.352 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29533 *rbd *testrbdnbd27356' 2024-09-17T07:48:26.352 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29533 *rbd *testrbdnbd27356' 2024-09-17T07:48:26.391 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:26.392 INFO:tasks.workunit.client.0.smithi098.stderr:+ break 2024-09-17T07:48:26.392 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:48:26.392 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29533 *rbd *testrbdnbd27356' 2024-09-17T07:48:26.392 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29533 *rbd *testrbdnbd27356' 2024-09-17T07:48:26.428 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:26.428+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:26.431 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:26.431 INFO:tasks.workunit.client.0.smithi098.stderr:+ QUIESCE_HOOK=/tmp/tmp.TyfTMF4aW5/quiesce.sh 2024-09-17T07:48:26.432 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.TyfTMF4aW5/quiesce.sh rbd/testrbdnbd27356 2024-09-17T07:48:26.432 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:48:26.432 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:48:26.433 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:48:26.433 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:48:26.435 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:48:26.435 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:48:26.435 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.TyfTMF4aW5/quiesce.sh rbd/testrbdnbd27356 2024-09-17T07:48:26.489 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:26.489 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:26.498 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:26.529 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:26.529+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:26.534 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:48:26.534 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:48:26.534 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:48:26.534 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:48:26.534 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:48:26.534 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:48:26.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29599 2024-09-17T07:48:26.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29599 2024-09-17T07:48:26.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29599 -C rbd-nbd 2024-09-17T07:48:26.598 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:48:26.598 INFO:tasks.workunit.client.0.smithi098.stdout: 29599 ? 00:00:00 rbd-nbd 2024-09-17T07:48:26.599 INFO:tasks.workunit.client.0.smithi098.stderr:+ test '!' -e /tmp/tmp.TyfTMF4aW5/quiesce.sh 2024-09-17T07:48:26.599 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false rbd snap create rbd/testrbdnbd27356@quiesce1 2024-09-17T07:48:26.599 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/testrbdnbd27356@quiesce1 2024-09-17T07:48:26.600 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:26.600 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:26.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:26.640 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete...2024-09-17T07:48:26.639+0000 7fdc0119f640 -1 librbd::ImageWatcher: 0x7fdbe8007a90 failed to notify quiesce: (1) Operation not permitted 2024-09-17T07:48:26.640 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:48:26.639+0000 7fdc0119f640 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-09-17T07:48:26.641 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete...failed. 2024-09-17T07:48:26.641 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-09-17T07:48:26.647 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:48:26.647 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:48:26.648 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:48:26.648 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:48:26.648 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:48:26.649 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:48:26.650 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:48:26.650 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:48:26.650 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:48:26.652 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:26.652+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:26.671 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:48:26.671 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:48:26.671 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112239 s, 93.4 MB/s 2024-09-17T07:48:26.672 INFO:tasks.workunit.client.0.smithi098.stderr:+ touch /tmp/tmp.TyfTMF4aW5/quiesce.sh 2024-09-17T07:48:26.672 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod +x /tmp/tmp.TyfTMF4aW5/quiesce.sh 2024-09-17T07:48:26.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-09-17T07:48:26.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/testrbdnbd27356@quiesce1 2024-09-17T07:48:26.715 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:26.715 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:26.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:26.768 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:26.768+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:26.822 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:26.823 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:26.831 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:26.831+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:26.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:26.931 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:26.931+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:26.970 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:26.970 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:26.979 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:27.063 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:27.063+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:27.117 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.117 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:27.132 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:27.132+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:27.233 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:27.233+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:27.263 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.264 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:27.334 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:27.334+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:27.410 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.410 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:27.461 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:27.461+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:27.515 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.516 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:27.568 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:27.568+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:27.637 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:27.637+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:27.638 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.639 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.648 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:27.738 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:27.738+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:27.791 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.791 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:27.839 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:27.840+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:27.901 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:27.901 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:27.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:27.940 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:27.940+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:28.011 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:28.011 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:28.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:28.063 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:28.063+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:28.121 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:28.121 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:28.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:28.170 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:28.170+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:28.243 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:28.243+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:28.271 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:28.271 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:28.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:28.317 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:48:28.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:28.360 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:28.360+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:28.382 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:28.382 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:28.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:28.444 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:28.444+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:28.465 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:28.482 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:28.482 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:28.489 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:28.489 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:28.497 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:28.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:28.513 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:28.529 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:28.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:28.545 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:28.545+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.431233893000002, 2024-09-17T07:48:28.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:28.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:28.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:28.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.423894559000001, 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:28.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:28.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.423177485, 2024-09-17T07:48:28.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:28.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:28.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:28.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:28.613 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.413949924000001, 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:28.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:28.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.404725356, 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:28.618 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.622 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.389250715999999, 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:28.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:28.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.627 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.379961206000001, 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:28.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:28.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.373370960999999, 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.632 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:28.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.636 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.357188480000001, 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:28.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.348113560000002, 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:28.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:28.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:28.645 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.007674684000001, 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:28.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:28.651 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.005622407000001, 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:28.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:28.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:28.657 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.945390755999998, 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:28.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 15.784211607, 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:28.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:28.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:28.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 15.784187306, 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.669 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.673 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 15.784152929999999, 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:28.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:28.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:28.679 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 15.784131573, 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:28.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 15.781462929, 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:28.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:28.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.689 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 15.771745356, 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.690 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:28.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.8056349669999996, 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:28.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:28.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:28.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:28.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:28.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:28.701 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:28.701 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:28.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:28.703 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:28.701+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:28.803 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:28.802+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:28.840 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:28.840 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:28.848 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:28.904 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:28.903+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:28.945 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:28.945 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:28.945 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:28.945 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:28.945 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:28.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:29.037 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:29.037+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:29.092 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.909888563999999, 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.093 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:29.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:29.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.097 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.901748525999999, 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:29.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.882125834, 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:29.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:29.106 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.874800602000001, 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:29.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.868188500999999, 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:29.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:29.116 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.864007178000001, 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:29.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:29.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.861694534000002, 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:29.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:29.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:29.126 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.836647429999999, 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:29.131 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.836309972999999, 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:29.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.502766545, 2024-09-17T07:48:29.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:29.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:29.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:29.140 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.408734548999998, 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:29.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.144 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.816503903000001, 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:29.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.810357034999999, 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:29.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:29.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.798050679999999, 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:29.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.522837417000002, 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:29.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.516823299999999, 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:29.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 20.511187073999999, 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:29.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:29.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.469867195999999, 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:29.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.3040215370000006, 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:29.176 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:29.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:29.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:29.179 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.2978394249999994, 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:29.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:29.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:29.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:29.184 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:29.186 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:29.185+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:29.186 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:29.286 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:29.286+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:29.299 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:29.299 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:29.308 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:29.311 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:48:29.387 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:29.387+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:29.411 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:29.412 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:29.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:29.488 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:29.488+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:29.525 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.358600622000001, 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.350941106000001, 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.342551537999999, 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:29.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:29.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.342180288000002, 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.334117323000001, 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.333746211000001, 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:29.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.324465805999999, 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:29.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:29.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.560 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.322806372999999, 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:29.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:29.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.316781548000002, 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:29.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.316531220999998, 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:29.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:29.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.309417383, 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:29.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.308900284, 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.581 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:29.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.289929117, 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:29.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.283672256999999, 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.277136195000001, 2024-09-17T07:48:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.904269785, 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:29.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:29.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.902309377999998, 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:29.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:29.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:29.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.016640742, 2024-09-17T07:48:29.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:29.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:29.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.705795104, 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:29.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:29.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:29.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 16.704433658999999, 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:29.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:29.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:29.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:29.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:29.634 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:29.635 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:29.636 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:29.634+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:29.736 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:29.736+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:29.778 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:29.778 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:29.778 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:29.778 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:29.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:29.837 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:29.837+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:29.888 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:29.888 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:29.888 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:29.888 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:29.888 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:29.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:29.940 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:29.940+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:29.944 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:48:29.945 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:29.945 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-17T07:48:29.945 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:48:29.945 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:48:29.945 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-17T07:48:29.999 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.740831494999998, 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:30.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.372525879000001, 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:30.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:30.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.680343623999999, 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:30.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:30.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.674486768000001, 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:30.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.662793809, 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:30.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:30.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:30.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.593939226, 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.028 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:30.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:30.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:30.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.582039937000001, 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.037 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:30.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.531195390000001, 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:30.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:30.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.525187183, 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:30.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:30.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:30.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.519486826000001, 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:30.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:30.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:30.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:30.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.513747201000001, 2024-09-17T07:48:30.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:30.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:30.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.180636152999998, 2024-09-17T07:48:30.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.073 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.079 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.335535256, 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.329448533000001, 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.245992232000001, 2024-09-17T07:48:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:30.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.239677800000001, 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:30.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:30.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:30.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:30.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.23330634, 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:30.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:30.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.227383864, 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:30.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:30.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:30.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:30.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:30.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.169400968, 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.211696296, 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:30.133 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:30.135 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:30.136 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:48:30.136 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:30.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:30.136 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:30.134+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:30.236 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:30.237 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:30.237 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:30.236+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:30.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:30.337 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:30.337+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:30.382 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:30.382 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:30.382 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:30.382 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:30.382 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:30.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:30.438 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:30.437+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:30.489 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:30.489 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:30.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:30.489 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:30.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.322897194999999, 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.490 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:30.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.315557860999998, 2024-09-17T07:48:30.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:30.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.314840787000001, 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:30.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:30.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:30.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.305613225999998, 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:30.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.296388658000001, 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:30.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:30.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.280914018000001, 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:30.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:30.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.271624507999999, 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:30.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:30.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:30.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.265034263, 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:30.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:30.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.248851781999999, 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:30.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.239776861999999, 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.536 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.899337985999999, 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.897285708999998, 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:30.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:30.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:30.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.837054058, 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:30.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.554 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.675874909000001, 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:30.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:30.559 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.675850608000001, 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:30.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:30.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.675816231999999, 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.565 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:30.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:30.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:30.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.675794875000001, 2024-09-17T07:48:30.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:30.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:30.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.575 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.673126231000001, 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:30.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.580 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.663408658000002, 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:30.581 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:30.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:30.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.586 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.697298269000001, 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:30.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:30.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:30.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.593 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:30.593 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:30.593 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:30.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:30.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:30.595 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:30.594+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:30.696 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:30.695+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:30.702 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:30.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:30.734 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:30.734 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:30.734 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:30.734 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:30.735 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:30.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:30.750 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:30.766 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:30.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:30.770 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:30.827 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:30.827+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:30.879 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:30.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:30.930 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:30.929+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:30.999 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:30.998+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:31.025 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.025 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:31.099 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:31.099+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:31.138 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.138 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.146 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:31.200 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:31.200+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:31.287 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.287 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:31.301 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:31.301+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:31.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:48:31.399 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.399 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.401 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:31.401+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:31.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:31.502 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:31.502+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:31.552 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.552 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:31.603 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:31.603+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:31.703 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.703 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.703 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:31.703+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:31.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:31.804 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:31.803+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:31.854 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.854 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:31.904 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:31.904+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:31.963 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:31.963 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:31.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:32.015 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:32.015+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:32.069 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.069 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:32.118 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:32.118+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:32.208 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:32.207+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:32.214 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.215 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:32.309 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:32.309+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:32.322 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:48:32.362 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.362 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:32.414 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:32.414+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:32.468 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.468 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:32.517 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:32.517+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:32.611 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:32.610+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:32.613 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.613 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:32.712 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:32.712+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:32.737 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.737 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:32.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:32.813 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:32.813+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:32.849 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:32.849 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:32.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:32.892 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:32.907 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:32.908 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:32.914 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:32.913+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:32.923 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:32.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:32.924 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:32.939 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:32.940 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:32.955 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:32.955 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:32.959 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:32.963 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.797126236, 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.964 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:32.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.968 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.78946672, 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.781077152000002, 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.974 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.979 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.780705902000001, 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:32.980 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.772642937000001, 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:32.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.772271825000001, 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:32.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:32.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.993 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.762991419999999, 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:32.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:32.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:32.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:32.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:32.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:32.999 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:32.999 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.761331986999998, 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:33.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.755307162000001, 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.755056835000001, 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:33.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.747942996999999, 2024-09-17T07:48:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.747425897999999, 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:33.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:33.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.027 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.728454730999999, 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:33.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:33.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:33.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:33.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.722197870999999, 2024-09-17T07:48:33.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:33.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.715661809, 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:33.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:33.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:33.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:33.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.342795399, 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.340834992000001, 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:33.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:33.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:33.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:33.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:33.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:33.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:33.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.062 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.062 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:33.062 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.455166355999999, 2024-09-17T07:48:33.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:33.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.062 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.063 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:33.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:33.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:33.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.144320717999999, 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:33.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:33.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:33.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.142959272999999, 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.078 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:33.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:33.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.084 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:33.085 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:33.085+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:33.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:33.186 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:33.186+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:33.231 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:33.231 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:33.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:33.287 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:33.286+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:33.340 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:33.341 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:33.341 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:33.341 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:33.341 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:33.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:33.390 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:33.390+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:33.488 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:33.488+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:33.494 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:33.494 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:33.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:33.494 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:33.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.495 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:33.495 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:33.495 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.234190905999998, 2024-09-17T07:48:33.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:33.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.496 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.496 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:33.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:33.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:33.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:33.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:33.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:33.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:33.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:33.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:33.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:33.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:33.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:33.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:33.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.865885290000001, 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:33.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:33.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:33.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:33.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.173703034999999, 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:33.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.167846179000001, 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.517 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.15615322, 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:33.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.087298637, 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:33.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.075399348000001, 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:33.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:33.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:33.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.024554801000001, 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:33.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:33.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:33.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.018546594, 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:33.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:33.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:33.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:33.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.012846237000002, 2024-09-17T07:48:33.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:33.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:33.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:33.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:33.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:33.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:33.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:33.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:33.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:33.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.568 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:33.568 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:33.568 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.007106612000001, 2024-09-17T07:48:33.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:33.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:33.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:33.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:33.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.570 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:33.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:33.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:33.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:33.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:33.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:33.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:33.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 20.673995563999998, 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:33.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:33.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.828894667, 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.822807944000001, 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.591 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.739351642999999, 2024-09-17T07:48:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.733037210999999, 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:33.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:33.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:33.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:33.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.726665750999999, 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.720743275, 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.662760379000002, 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.705055707, 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:33.636 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:33.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:33.638 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:33.637+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:33.738 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:33.738+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:33.741 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:33.741 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:33.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:33.839 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:33.839+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:33.848 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:33.848 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:33.848 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:33.848 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:33.848 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:33.856 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:33.940 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:33.940+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:33.955 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.788833908000001, 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.781494574, 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.780777499999999, 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.771549939, 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.762325370999999, 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.746850730999999, 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.737561221, 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.989 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.730970976000002, 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:33.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.714788495000001, 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:33.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.705713575000001, 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.365274699, 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.363222422, 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:34.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.302990771000001, 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:34.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.141811621999999, 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:34.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:34.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.024 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.141787320999999, 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.025 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:34.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:34.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.141752945, 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.141731587999999, 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.035 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:34.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.139062943999999, 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.040 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:34.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:34.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.129345370999999, 2024-09-17T07:48:34.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:34.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:34.050 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.163234982000001, 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:34.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:34.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:34.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.056 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:34.058 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:34.058 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:34.057+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:34.156 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:34.156 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:34.156 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:34.156 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:34.158 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:34.158+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:34.164 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:34.259 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:34.259+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:34.260 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:34.261 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:34.261 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:34.261 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:34.261 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:34.269 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:34.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:48:34.360 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:34.360+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.186414929999998, 2024-09-17T07:48:34.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:34.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:34.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.178274891999997, 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:34.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:34.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:34.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.158652199999999, 2024-09-17T07:48:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.151326967999999, 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:34.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.144714866999998, 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.140533544, 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.1382209, 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.113173795999998, 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.112836338999998, 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.779292910999999, 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.685260915000001, 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.093030269, 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.086883401000001, 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.074577046000002, 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.799363783, 2024-09-17T07:48:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.793349666000001, 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 25.787713440000001, 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.746393562000002, 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.580547902999999, 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.574365791, 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:34.459 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:34.461 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:34.460+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:34.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:34.562 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:34.562+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:34.564 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:34.564 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:34.564 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:34.564 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:34.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:34.662 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:34.662+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:34.674 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:34.675 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:34.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:34.763 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:34.763+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:34.827 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.660698150999998, 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.653038635000001, 2024-09-17T07:48:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.644649067000003, 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.644277817000003, 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.636214852000002, 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.635843739999999, 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.626563335, 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.624903902, 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.618879077000003, 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.618628749999999, 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.611514911999997, 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.610997812999997, 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:34.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.592026646000001, 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:34.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.585769786, 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:34.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:34.897 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 32.579233723999998, 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:34.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:34.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:34.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.206367314000001, 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:34.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:34.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.204406906999999, 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:34.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.318738271000001, 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.007892633000001, 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.006531188, 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:34.933 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:34.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:34.935 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:34.934+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:34.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:35.035 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:35.035+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:35.038 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:35.038 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:35.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:35.107 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:35.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:35.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:35.124 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:35.124 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:35.136 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:35.136+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:35.138 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:48:35.138 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:35.138 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T07:48:35.139 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.885334 2024-09-17T07:48:35.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 0.8853341224013972 2024-09-17T07:48:35.139 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:35.156 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:35.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:35.160 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.160 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:35.252 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:35.252+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:35.311 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.311 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.320 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:48:35.337 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:35.337+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:35.428 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.428 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:35.438 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:35.438+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:35.534 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.535 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.539 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:35.538+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:35.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:35.639 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:35.639+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:35.681 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.681 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.689 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:35.740 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:35.739+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:35.786 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.786 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:35.841 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:35.840+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:35.891 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:35.891 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:35.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:35.941 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:35.941+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:36.036 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.036 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.042 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:36.041+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:36.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:36.143 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:36.143+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:36.160 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.160 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:36.253 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:36.253+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:36.313 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.313 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:36.332 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.1 primary-affinity to 0.885334 (8580212) 2024-09-17T07:48:36.345 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:36.345+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:36.433 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.434 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.442 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:36.485 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:36.485+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:36.544 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.544 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.545 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:36.545+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:36.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:36.647 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:36.647+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:36.694 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.694 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:36.747 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:36.747+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:36.804 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.804 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.812 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:36.856 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:36.856+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:36.917 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:36.918 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:36.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:36.967 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:36.967+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:37.049 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:37.049+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:37.064 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:37.064 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:37.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:37.157 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:37.157+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:37.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:37.210 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:37.211 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:37.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:37.251 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:37.251+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:37.263 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:37.279 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:37.279 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:37.294 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:37.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:37.311 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:37.326 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:37.330 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:37.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:48:37.359 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:37.359 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:37.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:37.371 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:37.371+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:37.453 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:37.453+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:37.464 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:37.464 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:37.464 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:37.464 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:37.464 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:37.472 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:37.554 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:37.554+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.387978230999998, 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:37.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:37.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:37.575 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.379838192999998, 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:37.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:37.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:37.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.360215500999999, 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.352890269, 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:37.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.346278167999998, 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:37.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.342096845, 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:37.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.339784201000001, 2024-09-17T07:48:37.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:37.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:37.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.314737096999998, 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:37.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:37.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.314399639999998, 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:37.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:37.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.615 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.980856211999999, 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:37.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.886824216000001, 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.620 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:37.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.624 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.29459357, 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:37.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:37.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.288446702000002, 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.629 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:37.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:37.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:37.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.276140346999998, 2024-09-17T07:48:37.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.000927084000001, 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.994912967000001, 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.989276741000001, 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 21.947956863000002, 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.782111204, 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.775929091999998, 2024-09-17T07:48:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:37.664 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:37.665 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:37.665+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:37.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:37.766 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:37.765+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:37.767 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:37.767 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:37.767 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:37.767 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:37.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:37.866 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:37.866+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:37.877 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:37.877 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:37.877 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:37.885 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:37.970 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:37.970+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:38.029 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.862504688999998, 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:38.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.854845173000001, 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.846455605000003, 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:38.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.846084355000002, 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:38.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:38.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:38.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.838021390000002, 2024-09-17T07:48:38.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:38.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:38.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.837650277999998, 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:38.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:38.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:38.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.828369873, 2024-09-17T07:48:38.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:38.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:38.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.826710439999999, 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:38.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:38.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.820685615000002, 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:38.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.820435287999999, 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:38.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:38.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:38.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.076 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.813321449999997, 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:38.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:38.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.812804350999997, 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:38.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:38.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.793833184, 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:38.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:38.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.787576324, 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:38.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:38.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.096 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.781040261999998, 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:38.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:38.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:38.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.408173852000001, 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.102 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:38.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:38.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:38.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.406213444999999, 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.108 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:38.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:38.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:38.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.520544809, 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:38.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:38.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.209699171, 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:38.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.208337726, 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:38.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:38.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:38.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:38.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.131 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:38.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:38.133 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:38.132+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:38.233 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:38.233+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:38.275 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:38.275 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:38.275 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:38.275 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:38.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:38.334 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:38.334+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:48:38.387 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:38.387 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:38.387 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:38.387 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:38.387 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:38.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:38.436 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:38.436+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:38.536 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:38.536+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:38.539 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 36.281069383999998, 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.912763768000001, 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:38.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.220581512999999, 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.549 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.214724657000001, 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.203031698, 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.134177115, 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.122277826000001, 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.071433279000001, 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.065425072, 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.059724715000002, 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.053985090000001, 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 25.720874041999998, 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.875773145, 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.869686422000001, 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.786230120999999, 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.779915688999999, 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.773544228999999, 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.767621753, 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:38.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.709638857000002, 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.751934185, 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:38.656 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:38.658 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:38.658 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:38.657+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:38.759 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:38.758+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:38.763 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:38.763 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:38.763 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:38.763 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:38.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:38.859 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:38.859+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:38.869 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:38.869 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:38.869 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:38.869 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:38.869 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:38.878 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:38.960 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:38.960+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:38.976 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.810165904000002, 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:38.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:38.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:38.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:38.981 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.802826570000001, 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:38.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:38.985 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.802109496, 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:38.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:38.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.792881934999997, 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:38.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:38.995 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.783657367000004, 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:38.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:38.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:38.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.768182727000003, 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.000 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:39.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.758893217000001, 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:39.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.752302972000003, 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:39.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.736120491000001, 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:39.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.727045570999998, 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:39.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.386606695000001, 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:39.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.384554418, 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.029 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:39.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:39.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.035 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.324322767000002, 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:39.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:39.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.163143617999999, 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.163119317, 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.163084941000001, 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.163063584, 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.16039494, 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.150677367, 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.184566977999999, 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.080 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:39.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:39.082 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:39.081+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:39.179 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:39.179 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:39.182 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:39.182+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:39.187 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:39.284 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:39.283+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:39.284 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:39.285 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:39.285 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:39.285 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:39.285 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:39.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:39.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:39.385 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:39.384+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:39.433 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:39.433 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:39.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:39.433 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:39.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.250107858, 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:39.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:39.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.241967819999999, 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:39.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:39.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:39.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:39.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.222345128000001, 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:39.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:39.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.215019896000001, 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:39.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.451 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.208407794999999, 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:39.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.456 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.204226472000002, 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:39.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.201913828000002, 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:39.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.176866724, 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.176529266999999, 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:39.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.842985839000001, 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:39.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.748953842999999, 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:39.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.156723197000002, 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:39.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:39.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.150576329, 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.488 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.492 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.138269974, 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:39.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.863056710999999, 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:39.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.501 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.857042593999999, 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:39.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.851406367999999, 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:39.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.510 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.81008649, 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:39.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:39.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.644240831000001, 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:39.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.638058719, 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:39.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:39.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:39.523 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:39.524 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:39.525 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:39.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:39.526 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:39.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:39.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:39.527 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:39.527 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:39.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:39.528 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:39.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:39.529 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:39.524+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:39.626 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:39.625+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:39.671 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:39.671 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:39.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:39.726 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:39.726+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:39.781 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:39.781 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:39.789 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:39.827 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:39.827+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:39.890 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:39.891 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:39.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:39.943 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:39.943+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:40.001 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.001 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.009 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:40.029 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:40.029+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:40.130 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:40.130+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:40.150 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.150 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.159 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:40.231 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:40.231+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:40.301 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.302 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:40.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:48:40.354 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:40.354+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:40.412 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.412 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:40.433 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:40.433+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:40.534 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:40.534+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:40.559 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.559 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.567 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:40.635 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:40.635+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:40.666 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.666 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:40.736 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:40.736+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:40.812 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.812 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:40.837 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:40.836+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:40.916 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:40.917 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:40.925 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:40.968 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:40.968+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:41.023 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:41.023 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:41.031 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:41.075 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:41.075+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:41.137 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:41.137 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:41.139 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:41.140+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:41.146 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:41.240 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:41.240+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:41.288 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:41.288 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:41.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:41.341 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:41.341+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:41.344 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:48:41.346 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:48:41.347 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:41.347 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-17T07:48:41.347 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:48:41.347 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:48:41.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-09-17T07:48:41.400 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:41.400 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:41.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:41.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:41.688 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:41.687+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:41.689 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:41.689 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:48:41.689 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:41.788 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:41.788+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:41.793 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:41.793 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:41.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:41.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:41.823 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:41.823 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:41.823 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:41.840 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:41.840 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:41.855 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:41.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:41.900 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:41.900+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:41.904 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:41.905 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:41.905 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:41.905 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:41.905 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:41.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:41.997 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:41.997+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:42.057 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.798467617, 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.430162000999999, 2024-09-17T07:48:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.737979746000001, 2024-09-17T07:48:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.732122889999999, 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.720429930999998, 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.651575348000001, 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.639676059000003, 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.588831511999999, 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.582823304999998, 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.577122948000003, 2024-09-17T07:48:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 33.571383322999999, 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.238272275, 2024-09-17T07:48:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.393171378000002, 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.387084654999999, 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.303628354000001, 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.297313922000001, 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:42.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.290942462, 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:42.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:42.165 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.285019986000002, 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:42.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:42.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.22703709, 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:42.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:42.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:42.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:42.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 22.269332418000001, 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:42.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:42.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:42.183 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:42.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:42.185 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:42.184+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:42.285 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:42.284+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:42.291 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:42.291 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:42.291 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:42.291 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:42.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:42.385 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:42.385+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:42.398 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:42.398 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:42.398 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:42.398 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:42.398 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:42.407 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:42.491 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:42.491+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:42.545 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.379029019999997, 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.371689686000003, 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.370972612000003, 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.361745051, 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.352520482999999, 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.337045842999999, 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.327756333000004, 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.321166087999998, 2024-09-17T07:48:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.304983606999997, 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.295908687000001, 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.955469811, 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.953417534000003, 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.893185883000001, 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.732006733999999, 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:42.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:42.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:42.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.731982432999999, 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:42.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.630 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.731948057, 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:42.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:42.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:42.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:42.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.731926699999999, 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:42.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.729258055999999, 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:42.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:42.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:42.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.719540482999999, 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.649 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:42.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:42.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.753430093999999, 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:42.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:42.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:42.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.661 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:42.662 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:42.662 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:42.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:42.664 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:42.663+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:42.763 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:42.763+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:42.801 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:42.801 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:42.801 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:42.801 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:42.809 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:42.864 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:42.864+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:42.906 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:42.906 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:42.906 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:42.906 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:42.906 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:42.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:42.965 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:42.965+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:43.013 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:43.013 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:43.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:43.013 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:43.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.013 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.830279709000003, 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:43.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.822139671000002, 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.018 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:43.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:43.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:43.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:43.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.802516979000004, 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:43.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:43.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:43.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.795191746999997, 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:43.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.788579646000002, 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:43.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.784398322999998, 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.037 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:43.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.782085678999998, 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:43.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.757038575000003, 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:43.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:43.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.756701118000002, 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.052 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:43.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.056 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.423157689999996, 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:43.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.060 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.329125693999998, 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:43.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:43.064 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.736895048000001, 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:43.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.068 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.730748179999999, 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.718441824999999, 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:43.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:43.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.443228562000002, 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:43.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:43.082 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.437214445000002, 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:43.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:43.086 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 34.431578219000002, 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:43.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.390258340999999, 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:43.091 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:43.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.095 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.224412682000001, 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:43.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.218230569999999, 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:43.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:43.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:43.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:43.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:43.104 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:43.105 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:43.105+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:43.106 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:43.206 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:43.206+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:43.220 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:43.221 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:43.221 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:43.221 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:43.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:43.306 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:43.307+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:43.332 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:43.332 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:43.333 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:43.333 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:43.333 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:43.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:43.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:48:43.428 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:43.428+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:43.445 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.278934720000002, 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.271275203999998, 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.262885636, 2024-09-17T07:48:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.262514385999999, 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.254451420999999, 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.254080309000003, 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.244799903999997, 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.243140470999997, 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.237115645999999, 2024-09-17T07:48:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.236865319000003, 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.229751481000001, 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.229234382000001, 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.210263214999998, 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.204006354999997, 2024-09-17T07:48:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.197470293000002, 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:43.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:43.519 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.824603883000002, 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:43.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:43.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.525 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.822643476000003, 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.936974839999998, 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.531 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:43.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:43.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:43.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.626129202000001, 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:43.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:43.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:43.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:43.542 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.624767757000001, 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:43.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:43.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:43.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.549 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:43.550 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:43.551 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:43.550+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:43.651 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:43.651+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:43.693 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:43.693 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:43.693 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:43.693 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:43.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:43.752 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:43.752+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:43.843 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:43.843 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:43.843 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:43.843 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:43.843 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:43.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:43.852 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:43.852+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:43.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:43.953 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:43.953+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.696669331999999, 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 36.328363715999998, 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.636181461, 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.630324604999998, 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.618631645999997, 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.549777063000001, 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.537877774000002, 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.487033226999998, 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.481025019999997, 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.475324663000002, 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.469585037999998, 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.136473989999999, 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.291373093000001, 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.285286370000001, 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.201830069, 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.195515637, 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.189144176999999, 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.183221701000001, 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.125238804999999, 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.167534133, 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:44.070 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:44.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:44.072 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:44.073 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:44.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:44.073 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:44.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:44.074 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:44.074 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:44.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:44.076 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:44.071+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:44.172 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:44.172+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:44.184 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:44.185 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:44.194 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:44.278 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:44.278+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:44.334 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:44.334 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:44.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:44.374 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:44.374+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:44.475 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:44.475+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:44.480 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:44.480 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:44.487 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:48:44.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:44.576 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:44.576+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:44.587 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:44.587 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:44.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:44.680 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:44.679+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:44.733 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:44.733 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:44.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:44.778 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:44.777+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:44.878 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:44.877+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:44.879 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:44.880 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:44.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:44.982 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:44.981+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:45.003 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.003 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:45.083 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:45.082+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:45.114 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.114 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.123 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:45.207 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:45.207+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:45.265 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.265 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.273 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:45.314 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:45.314+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:45.388 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:45.386+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:45.415 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.415 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:45.508 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:45.508+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:45.568 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.568 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:45.618 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:45.617+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:45.688 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:45.688+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:45.718 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.718 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:45.789 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:45.788+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:45.835 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.835 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:45.890 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:45.890+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:45.982 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:45.982 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:45.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:45.991 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:45.989+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:46.076 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:46.091 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:46.090+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:46.128 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:46.129 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:46.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:46.196 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:46.196 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:46.212 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:46.212 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:46.212 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:46.220 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:46.219+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:46.228 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:46.228 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:46.243 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:46.243 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:46.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:46.275 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:46.276 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:46.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:46.293 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:46.293+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:46.394 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:46.394+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:46.423 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:46.423 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:46.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:46.495 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:46.493+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:46.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:48:46.533 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.349256849, 2024-09-17T07:48:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.341116810999999, 2024-09-17T07:48:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.321494119, 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.557 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.314168887000001, 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:46.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.307556785999999, 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.566 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.303375463000002, 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:46.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.571 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.301062819000002, 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:46.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:46.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.276015715, 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.577 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:46.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:46.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:46.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.275678257999999, 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.584 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:46.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:46.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.942134830000001, 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.848102834000002, 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:46.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:46.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.255872187999998, 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:46.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:46.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.249725320000003, 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:46.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.237418965000003, 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.962205701999999, 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.620 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.956191584999999, 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:46.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.950555358999999, 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.909235481, 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:46.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.743389822000001, 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:46.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.73720771, 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:46.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:46.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:46.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:46.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:46.642 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:46.644 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:46.642+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:46.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:46.690 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:48:46.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:46.690 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-17T07:48:46.690 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:48:46.690 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:48:46.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-09-17T07:48:46.744 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:46.743+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:46.746 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:46.746 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:46.747 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:46.747 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:46.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:46.789 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:46.789 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "" 2024-09-17T07:48:46.789 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:46.845 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:46.844+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:46.855 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:46.856 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:46.864 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:46.946 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:46.945+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.800330484, 2024-09-17T07:48:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:46.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.792670968000003, 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.784281399999998, 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:46.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:46.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:46.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:46.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.783910149999997, 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.775847185000003, 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.775476073, 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.766195668000002, 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:47.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:47.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.764536235000001, 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:47.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:47.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.758511409999997, 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:47.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:47.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:47.012 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.758261083000001, 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:47.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:47.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.751147244999999, 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:47.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:47.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.750630145999999, 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.731658979000002, 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:47.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:47.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:47.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.725402119000002, 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:47.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:47.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.718866057, 2024-09-17T07:48:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:47.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:47.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.345999646999999, 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:47.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:47.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:47.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.344039240000001, 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:47.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:47.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:47.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:47.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:47.058 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 38.458370604000002, 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:47.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:47.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.064 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.147524965999999, 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.146163520999998, 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:47.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:47.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.077 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:47.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:47.079 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:47.077+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:47.179 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:47.179+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:47.222 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:47.222 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:47.222 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:47.222 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:47.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:47.280 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:47.279+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:47.331 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:47.331 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:47.331 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:47.332 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:47.332 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:47.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:47.380 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:47.379+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:47.482 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:47.481+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:47.483 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:47.483 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:47.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:47.483 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.224648979999998, 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:47.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.856343363999997, 2024-09-17T07:48:47.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:47.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.164161108999998, 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.158304252999997, 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.146611294000003, 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.077756710999999, 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.065857422000001, 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.015012874999996, 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.009004668000003, 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.003304311000001, 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 38.997564685999997, 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.664453637999998, 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.819352740999999, 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.813266018, 2024-09-17T07:48:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.729809716999998, 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.723495284999998, 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.717123825000002, 2024-09-17T07:48:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.711201349, 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.653218453000001, 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.695513780999999, 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:47.594 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:47.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:47.596 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:48:47.597 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:47.595+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:47.696 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:47.695+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:47.699 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:47.699 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:47.699 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:47.699 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:47.707 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:47.797 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:47.797+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:47.845 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:47.845 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:47.845 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:47.845 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:47.845 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:47.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:47.898 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:47.897+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:47.952 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:47.952 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:47.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:47.952 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:47.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.952 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.785543640999997, 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:47.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.778204307000003, 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:47.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:47.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.777487233000002, 2024-09-17T07:48:47.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:47.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:47.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:47.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:47.966 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.768259671999999, 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:47.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.759035103999999, 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.975 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.743560463999998, 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:47.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:47.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.734270954000003, 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.984 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.727680708999998, 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:47.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.711498227999996, 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:47.989 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.702423308, 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:47.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.361984432, 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:47.998 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:47.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:48.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:48.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.359932155000003, 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:48.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.299700504, 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:48.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:48.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.138521355000002, 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:48.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.138497053999998, 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:48.021 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:48.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.026 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.138462678000003, 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:48.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:48.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.138441321000002, 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:48.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.135772676999999, 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:48.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:48.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.126055104000002, 2024-09-17T07:48:48.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.159944715000002, 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:48.048 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:48.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:48.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.054 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:48.054 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:48.054 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:48.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:48.056 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:48.054+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:48.156 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:48.155+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:48.194 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:48.195 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:48.195 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:48.195 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:48.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:48.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:48.287 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:48.286+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:48.340 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:48.340 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:48.340 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:48.340 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:48.340 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:48.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:48.357 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:48.373 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:48.373 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:48.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:48.389 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:48.389 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:48.390 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:48.388+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:48.405 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:48.405 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:48.421 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:48.425 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:48.458 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:48.457+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:48.486 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:48.486 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:48.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:48.559 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:48.558+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:48.597 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:48.597 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:48.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:48.660 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:48.660+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:48.707 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:48.707 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:48.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:48.761 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:48.761+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:48.857 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:48.857 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:48.862 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:48.862+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:48.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:48.964 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:48.963+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:49.007 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.007 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:49.064 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:49.064+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:49.117 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.117 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:49.165 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:49.164+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:49.266 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:49.266+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:49.267 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.267 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.275 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:49.367 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:49.365+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:49.384 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.384 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.393 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:49.468 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:49.467+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:49.531 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.532 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:49.568 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:49.568+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:49.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:48:49.638 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.638 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.647 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:49.669 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:49.669+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:49.745 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.746 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.754 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:49.770 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:49.770+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:49.852 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.852 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:49.904 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:49.903+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:49.958 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:49.958 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:49.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:50.007 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:50.006+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:50.069 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:50.069 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:50.074 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:50.073+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:50.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:50.175 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:50.175+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:50.178 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:50.179 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:50.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:50.277 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:50.276+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:50.329 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:50.329 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:50.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:50.378 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:50.377+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:50.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:50.479 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:50.478+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:50.481 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:50.481 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:50.489 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:50.494 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:48:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:50.561 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:50.562 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:50.578 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:50.578 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:50.578 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:50.579 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:50.578+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:50.594 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:50.609 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:50.610 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:50.610 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:48:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:48:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:50.613 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:50.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:50.702 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:50.702+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:50.762 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:50.762 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:50.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.503430709, 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:50.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:50.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 43.135125092999999, 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:50.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.442942838, 2024-09-17T07:48:50.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:50.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.437085981999999, 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.777 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:50.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.425393022999998, 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.783 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:50.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:50.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.356538440000001, 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.789 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.344639151000003, 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.293794603999999, 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.287786396999998, 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.282086040000003, 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:50.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:50.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.276346414999999, 2024-09-17T07:48:50.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:50.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:50.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:50.824 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.943235367, 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:50.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:50.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:50.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.098134469999998, 2024-09-17T07:48:50.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.092047747000002, 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.008591445999997, 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:50.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.002277014000001, 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:50.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:50.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.995905553999997, 2024-09-17T07:48:50.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:50.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:50.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.989983078000002, 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.859 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:50.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:50.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:50.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 34.932000182000003, 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:50.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:50.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 30.974295510000001, 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:50.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:50.874 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:50.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:50.876 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:50.875+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:50.977 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:50.977+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:50.981 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:50.981 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:50.981 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:50.981 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:50.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:51.078 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:51.077+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:51.088 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:51.089 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:51.089 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:51.089 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:51.089 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:51.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:51.179 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:51.179+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:51.236 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:51.236 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:51.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.069166291000002, 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:51.237 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:51.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.061826957000001, 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:51.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:51.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:51.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:51.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:51.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:51.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:51.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:51.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.061109883, 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:51.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:51.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:51.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.051882321999997, 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.255 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:51.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:51.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:51.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.260 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.042657753999997, 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:51.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.027183114000003, 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:51.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:51.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.017893604000001, 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.011303359000003, 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.995120878000002, 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.986045957999998, 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:51.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:51.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:51.292 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.645607081999998, 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:51.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:51.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:51.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.299 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.300 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.643554805000001, 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:51.301 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:51.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.583323153999999, 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.307 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:51.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.422144005, 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.314 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:51.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.319 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.422119703999996, 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:51.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:51.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.422085328000001, 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.422063971, 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.419395326999997, 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:51.343 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 38.409677754000001, 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:51.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:51.348 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.443567365, 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:51.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:51.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.354 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:51.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:48:51.356 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:51.355+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:51.456 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:51.456 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:51.456 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:51.456 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:51.456 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:51.455+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:51.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:48:51.557 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:51.557+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:51.602 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:51.602 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:51.602 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:51.602 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:51.602 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:51.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:48:51.658 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:51.658+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:51.710 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:51.710 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:51.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:51.710 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:51.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:33 1.3 e45/12 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head v 45'7, mlcod=45'5)", 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.526693487999999, 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013167029, 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181703+0000", 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.181712+0000", 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6309999999999998e-06 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189811+0000", 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080990790000000003 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189827+0000", 2024-09-17T07:48:51.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5517999999999999e-05 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189833+0000", 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9410000000000004e-06 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189876+0000", 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3015000000000001e-05 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189900+0000", 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3992999999999999e-05 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194829+0000", 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004929063 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.194870+0000", 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0788999999999999e-05 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:34 1.3 e45/12 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head v 45'8, mlcod=45'5)", 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.518553449999999, 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014282471, 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189843+0000", 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.15e-07 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198615+0000", 2024-09-17T07:48:51.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.008771437 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198639+0000", 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3912000000000001e-05 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198647+0000", 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198714+0000", 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6692000000000005e-05 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198748+0000", 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4007999999999997e-05 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204083+0000", 2024-09-17T07:48:51.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0053346289999999996 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204125+0000", 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2478000000000002e-05 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:44 1.4 e45/29 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.498930758, 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012829224, 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:51.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209465+0000", 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.209467+0000", 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.748e-06 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216393+0000", 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069255000000000002 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216411+0000", 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8593e-05 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216422+0000", 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0801999999999999e-05 2024-09-17T07:48:51.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216470+0000", 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.808e-05 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216494+0000", 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589999999999999e-05 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222239+0000", 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057454649999999999 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.222295+0000", 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5445999999999997e-05 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:47 1.4 e45/29 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head v 45'6, mlcod=45'3)", 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:51.727 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.491605526000001, 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.019543522000000001, 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216791+0000", 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216792+0000", 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2500000000000004e-07 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227542+0000", 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010750707 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227555+0000", 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2814e-05 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227565+0000", 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.6050000000000004e-06 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227609+0000", 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4601000000000002e-05 2024-09-17T07:48:51.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227632+0000", 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2745e-05 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236293+0000", 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0086611789999999998 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.236334+0000", 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0945999999999997e-05 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 1.1 1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.484993424999999, 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017684238000000001, 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.732 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223403+0000", 2024-09-17T07:48:51.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223404+0000", 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5799999999999999e-06 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224933+0000", 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015286309999999999 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224959+0000", 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6071000000000002e-05 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.224967+0000", 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225024+0000", 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.736e-05 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.225185+0000", 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000160774 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.240963+0000", 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015778487000000001 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241018+0000", 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5104e-05 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241087+0000", 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8703999999999998e-05 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:48 1.3 e45/12 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head v 45'9, mlcod=45'7)", 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.480812102000002, 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016052154999999999, 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.738 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227584+0000", 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.227585+0000", 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9300000000000006e-07 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235789+0000", 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082037849999999999 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235806+0000", 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7028999999999999e-05 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235812+0000", 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1190000000000002e-06 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235853+0000", 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.083e-05 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.235880+0000", 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7310000000000001e-05 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243567+0000", 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0076864350000000001 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243636+0000", 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9653999999999994e-05 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 1.1 1:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.478499458000002, 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014098701, 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229897+0000", 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229899+0000", 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1600000000000001e-06 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233462+0000", 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003562871 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233481+0000", 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8677000000000001e-05 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233489+0000", 2024-09-17T07:48:51.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2379999999999997e-06 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233549+0000", 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0340000000000002e-05 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.233816+0000", 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026696700000000001 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243890+0000", 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010073976 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243935+0000", 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4620000000000003e-05 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.243996+0000", 2024-09-17T07:48:51.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0851999999999999e-05 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.453452354, 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010965353000000001, 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254944+0000", 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254946+0000", 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.272e-06 2024-09-17T07:48:51.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256766+0000", 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001819607 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256788+0000", 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1755000000000001e-05 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256796+0000", 2024-09-17T07:48:51.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8289999999999997e-06 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256856+0000", 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000002e-05 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.256980+0000", 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124423 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.257095+0000", 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114459 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.263001+0000", 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.005906082 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265854+0000", 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002853225 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265886+0000", 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1742000000000002e-05 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265909+0000", 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3589000000000001e-05 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4801.0:57 1.3 e45/12 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head v 45'10, mlcod=45'8)", 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.453114896999999, 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090836379999999998, 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255281+0000", 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.255282+0000", 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:48:51.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259702+0000", 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004419966 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259723+0000", 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.1282000000000001e-05 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259732+0000", 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3040000000000005e-06 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259773+0000", 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1758999999999998e-05 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.259791+0000", 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.802e-05 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264335+0000", 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045431170000000002 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264365+0000", 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0414999999999999e-05 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:56 1.7 1:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:51.759 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.119571469, 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003254977, 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588825+0000", 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8400000000000005e-07 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588829+0000", 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.382e-06 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588836+0000", 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1119999999999998e-06 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588840+0000", 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8569999999999997e-06 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589158+0000", 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000318735 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.589202+0000", 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3661000000000001e-05 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.592080+0000", 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002877746 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:63 1.7 1:fe4db0d1:::rbd_data.123eb9c2b614.000000000000000e:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.025539473000002, 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042008599999999998, 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682857+0000", 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9299999999999998e-07 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682861+0000", 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9870000000000001e-06 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682870+0000", 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8529999999999998e-06 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.682874+0000", 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8279999999999996e-06 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683527+0000", 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065335000000000002 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.683597+0000", 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0038999999999998e-05 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.687058+0000", 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034603099999999999 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.768 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:612 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'9, mlcod=49'7)", 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.433308826999998, 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022588220000000002, 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275087+0000", 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275088+0000", 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3499999999999996e-07 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275949+0000", 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000861034 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275960+0000", 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0733e-05 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275964+0000", 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2169999999999996e-06 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.275991+0000", 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6701999999999999e-05 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.276002+0000", 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.132e-05 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277310+0000", 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001307622 2024-09-17T07:48:51.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.277346+0000", 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6659000000000003e-05 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:614 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'11, mlcod=49'9)", 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.427161959000003, 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022367760000000002, 2024-09-17T07:48:51.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281234+0000", 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.281235+0000", 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6400000000000003e-07 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282062+0000", 2024-09-17T07:48:51.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082722700000000004 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282072+0000", 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0166000000000001e-05 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282076+0000", 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8909999999999997e-06 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282103+0000", 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6458e-05 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.282115+0000", 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2483999999999999e-05 2024-09-17T07:48:51.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283434+0000", 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001319448 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.283471+0000", 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6637999999999997e-05 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.778 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:618 3.2 e49/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 49'15, mlcod=49'13)", 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.414855604000003, 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0024911059999999999, 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:51.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.293541+0000", 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0400000000000002e-07 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294364+0000", 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082269699999999995 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294372+0000", 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1079999999999993e-06 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294375+0000", 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5470000000000001e-06 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294406+0000", 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0552000000000002e-05 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.294422+0000", 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6140000000000001e-05 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296008+0000", 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001585695 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.296032+0000", 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3963000000000001e-05 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.783 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:724 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'18, mlcod=49'16)", 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.139642340999998, 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026261470000000001, 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568754+0000", 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.568755+0000", 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4899999999999995e-07 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569973+0000", 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012182880000000001 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569985+0000", 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2439999999999999e-05 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.569990+0000", 2024-09-17T07:48:51.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2690000000000001e-06 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570015+0000", 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5191e-05 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.570025+0000", 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0477e-05 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571357+0000", 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013314449999999999 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.571380+0000", 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3388e-05 2024-09-17T07:48:51.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.789 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.789 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:726 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'20, mlcod=49'18)", 2024-09-17T07:48:51.789 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:51.790 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.133628223999999, 2024-09-17T07:48:51.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021397489999999998, 2024-09-17T07:48:51.790 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.790 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.790 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:51.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574768+0000", 2024-09-17T07:48:51.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.574769+0000", 2024-09-17T07:48:51.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7700000000000004e-07 2024-09-17T07:48:51.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575558+0000", 2024-09-17T07:48:51.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078887399999999996 2024-09-17T07:48:51.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575569+0000", 2024-09-17T07:48:51.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1021e-05 2024-09-17T07:48:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575572+0000", 2024-09-17T07:48:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.658e-06 2024-09-17T07:48:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575616+0000", 2024-09-17T07:48:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3387000000000003e-05 2024-09-17T07:48:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.575626+0000", 2024-09-17T07:48:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.028e-05 2024-09-17T07:48:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576886+0000", 2024-09-17T07:48:51.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012596809999999999 2024-09-17T07:48:51.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.576908+0000", 2024-09-17T07:48:51.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2171000000000002e-05 2024-09-17T07:48:51.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4950.0:728 3.2 e49/16 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head v 49'22, mlcod=49'20)", 2024-09-17T07:48:51.800 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:51.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.127991997999999, 2024-09-17T07:48:51.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021395680000000001, 2024-09-17T07:48:51.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.801 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:51.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580404+0000", 2024-09-17T07:48:51.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.580405+0000", 2024-09-17T07:48:51.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0100000000000005e-07 2024-09-17T07:48:51.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581213+0000", 2024-09-17T07:48:51.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080846100000000001 2024-09-17T07:48:51.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581224+0000", 2024-09-17T07:48:51.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0993e-05 2024-09-17T07:48:51.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581228+0000", 2024-09-17T07:48:51.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7859999999999998e-06 2024-09-17T07:48:51.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581271+0000", 2024-09-17T07:48:51.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2864999999999997e-05 2024-09-17T07:48:51.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.581282+0000", 2024-09-17T07:48:51.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1066e-05 2024-09-17T07:48:51.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582515+0000", 2024-09-17T07:48:51.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012326170000000001 2024-09-17T07:48:51.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:08.582544+0000", 2024-09-17T07:48:51.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9179e-05 2024-09-17T07:48:51.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.811 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.811 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4886.0:44 3.2 e53/16 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:head v 53'30, mlcod=52'28)", 2024-09-17T07:48:51.811 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:51.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.086672120000003, 2024-09-17T07:48:51.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002401107, 2024-09-17T07:48:51.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.812 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:51.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621724+0000", 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.621725+0000", 2024-09-17T07:48:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:48:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622782+0000", 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001057716 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622794+0000", 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1239e-05 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622799+0000", 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622832+0000", 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2814000000000001e-05 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.622847+0000", 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5294e-05 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624099+0000", 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012520789999999999 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:15.624125+0000", 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6298000000000001e-05 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(osd.4.11:77 1.7 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.920826461000001, 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0035158640000000001, 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:51.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787570+0000", 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1600000000000001e-07 2024-09-17T07:48:51.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787574+0000", 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4479999999999999e-06 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787581+0000", 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787586+0000", 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.787960+0000", 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00037407299999999999 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.788034+0000", 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3845999999999997e-05 2024-09-17T07:48:51.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.791086+0000", 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030517360000000002 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:223 3.2 e55/16 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head v 55'45, mlcod=53'43)", 2024-09-17T07:48:51.823 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.914644349, 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010249958, 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:51.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793752+0000", 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.793754+0000", 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0150000000000002e-06 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798578+0000", 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004824264 2024-09-17T07:48:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798604+0000", 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6191e-05 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798616+0000", 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1921e-05 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798659+0000", 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2744000000000003e-05 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.798680+0000", 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0721e-05 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803946+0000", 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052659020000000003 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:51.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804002+0000", 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6199999999999997e-05 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:51.828 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:51.829 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:48:51.829 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:48:51.829 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [1, 2, 4, 5] 2024-09-17T07:48:51.829 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T07:48:51.829 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:48:51.830 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:48:51.829+0000 7f16893f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31602) UID: 0 2024-09-17T07:48:51.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:51.856 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T07:48:51.856 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-17T07:48:51.856 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:48:51.856 INFO:tasks.thrashosds.thrasher:remote for exp osd 3 is ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:48:51.856 DEBUG:teuthology.orchestra.run.smithi094:> type ceph-objectstore-tool 2024-09-17T07:48:51.910 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T07:48:51.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-17T07:48:51.930 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:51.930+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:51.946 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:51.946 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:51.946 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:51.946 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:51.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:52.040 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:52.040+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:52.101 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:52.101 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:52.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:52.151 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:52.150+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:52.232 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:52.232+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.086473216000002, 2024-09-17T07:48:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:52.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.078813699999998, 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:52.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:52.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:52.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.263 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.070424131999999, 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.070052881999999, 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:52.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:52.274 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.061989916999998, 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.061618805000002, 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:52.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:52.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:52.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.052338399999996, 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:52.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.050678967000003, 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.044654141999999, 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:52.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:52.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.044403815000003, 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:52.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.037289977, 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:52.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:52.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:52.312 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.036772878000001, 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.313 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:52.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:52.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.017801710999997, 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.319 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:52.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.011544850999996, 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.324 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:52.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:52.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:52.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:52.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:52.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:52.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:52.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.331 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:52.331 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:52.331 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.005008789000001, 2024-09-17T07:48:52.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:52.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.332 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.332 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:52.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:52.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:52.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:52.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:52.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:52.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:52.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:52.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:52.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:52.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.632142379000001, 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:52.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:52.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.630181972000003, 2024-09-17T07:48:52.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:52.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:52.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:52.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:52.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:52.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 43.744513335999997, 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.356 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:52.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:52.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:52.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:52.364 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.433667698000001, 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:52.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.432306253, 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:52.371 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:52.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:52.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:52.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:52.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:52.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:52.378 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:52.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:52.380 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:52.379+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:52.481 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:52.479+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:52.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:48:52.526 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:52.526 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:52.526 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:52.526 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:52.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:52.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:52.619 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:52.619+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:52.682 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:52.682+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:52.683 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:52.684 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:52.684 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:52.684 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:52.684 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:52.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:52.740 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:52.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:52.757 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:52.757 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:52.757 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:52.773 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:52.773 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:52.773 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:48:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:52.776 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:52.783 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:52.782+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:52.827 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:52.827 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:52.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:52.883 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:52.883+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:52.934 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:52.934 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:52.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:52.986 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:52.985+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:53.036 INFO:teuthology.orchestra.run.smithi094.stdout:2.0 2024-09-17T07:48:53.037 INFO:teuthology.orchestra.run.smithi094.stdout:1.6 2024-09-17T07:48:53.037 INFO:teuthology.orchestra.run.smithi094.stdout:1.0 2024-09-17T07:48:53.043 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.043 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.052 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:53.085 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:53.085+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:53.186 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:53.185+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:53.190 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.190 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.198 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:53.287 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:53.287+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:53.304 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.304 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:53.388 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:53.388+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:53.414 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.414 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:53.489 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:53.488+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:53.507 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:48:53.527 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.527 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:53.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 1.6 --file /var/log/ceph/exp.1.6.3 2024-09-17T07:48:53.621 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:53.620+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:53.678 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.679 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:53.729 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:53.728+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:53.792 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:53.792+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:53.829 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.829 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:53.921 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:53.921+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:53.980 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:53.980 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:53.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:54.029 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.029+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.087 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.088 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.095 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:54.093+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:54.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:54.195 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.195+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.233 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.233 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:54.296 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.296+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.338 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.339 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:54.398 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.397+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.452 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.452 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:54.501 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.501+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.599 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:54.599+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:54.601 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.601 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:54.687 INFO:teuthology.orchestra.run.smithi094.stderr:Exporting 1.6 info 1.6( empty local-lis/les=29/30 n=0 ec=12/12 lis/c=29/29 les/c/f=30/30/0 sis=29) 2024-09-17T07:48:54.693 INFO:teuthology.orchestra.run.smithi094.stderr:Export successful 2024-09-17T07:48:54.700 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.700+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.752 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.752 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:54.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:54.802 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:54.801+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:54.902 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:54.901+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:54.903 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:54.905 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:54.906 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:54.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:54.919 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:54.920 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:54.937 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:54.937 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:54.937 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:54.953 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:54.954 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:54.954 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:48:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:54.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:55.003 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:55.002+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:55.022 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:55.022 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:55.022 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:55.022 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:55.022 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:55.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:55.114 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:55.114+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:55.174 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.915201166999999, 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:55.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.546895550999999, 2024-09-17T07:48:55.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.854713296, 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.184 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:55.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.848856439999999, 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:55.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.195 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.837163480999997, 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:55.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:55.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.768308898000001, 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.756409609000002, 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.705565061999998, 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.699556854999997, 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.693856498000002, 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:55.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:55.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:55.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:55.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:55.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.688116872999998, 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:55.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:55.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 42.355005824999999, 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.509904927999997, 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.503818205000002, 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.420361904000004, 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.414047472, 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.407676012000003, 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.401753536000001, 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.343770640000002, 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.386065967999997, 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:55.291 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:55.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 1.6 2024-09-17T07:48:55.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:55.294 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:55.292+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:55.394 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:55.394+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:55.435 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:55.435 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:55.435 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:55.435 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:55.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:55.495 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:55.494+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:55.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:48:55.541 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:55.541 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:55.542 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:55.542 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:55.550 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:55.596 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:55.595+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:55.696 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:55.696+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:55.710 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.543547701000001, 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.536208367, 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.535491293, 2024-09-17T07:48:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.526263731999997, 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.517039164000003, 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.501564524000003, 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.492275014000001, 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.485684769000002, 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.469502288000001, 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.460427367999998, 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.119988491999997, 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.117936215, 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.057704563999998, 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.896525414999999, 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.896501114000003, 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.896466738000001, 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.896445380999999, 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.893776737000003, 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 42.884059164, 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.917948774999999, 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:55.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:55.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:55.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:55.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:55.814 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:55.815 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:55.814+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:55.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:55.915 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:55.915+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:55.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:55.917 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:55.917 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:55.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:56.016 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:56.016+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:56.067 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:56.067 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:56.067 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:56.067 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:56.067 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:56.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:56.117 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:56.116+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:56.217 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:56.217+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.051951615999997, 2024-09-17T07:48:56.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:56.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:48:56.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.0442921, 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:48:56.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:48:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:48:56.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.035902532000001, 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:48:56.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:48:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.035531282000001, 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:48:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:48:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.027468317, 2024-09-17T07:48:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:48:56.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:48:56.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.027097204999997, 2024-09-17T07:48:56.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.250 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.017816799999999, 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:48:56.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:48:56.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.016157366999998, 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.010132542000001, 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.261 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:48:56.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.009882214999998, 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:56.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:48:56.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:56.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.002768377000002, 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:48:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:48:56.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:56.278 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.002251278000003, 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:48:56.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:48:56.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.983280110999999, 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:48:56.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.977023250999999, 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:48:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:48:56.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.970487188999996, 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:48:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:48:56.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.597620779000003, 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:48:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:48:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:48:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.595660371999998, 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:48:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:48:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:48:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.709991735999999, 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:48:56.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:48:56.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 43.399146098000003, 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:48:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:48:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 43.397784653000002, 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.329 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:48:56.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:48:56.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:48:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:48:56.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:48:56.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.339 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:56.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:56.341 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:56.339+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:56.422 INFO:teuthology.orchestra.run.smithi094.stdout: marking collection for removal 2024-09-17T07:48:56.422 INFO:teuthology.orchestra.run.smithi094.stdout:setting '_remove' omap key 2024-09-17T07:48:56.422 INFO:teuthology.orchestra.run.smithi094.stdout:finish_remove_pgs 1.6_head removing 1.6 2024-09-17T07:48:56.423 INFO:teuthology.orchestra.run.smithi094.stdout:Remove successful 2024-09-17T07:48:56.441 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:56.440+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:56.483 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:56.484 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:56.484 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:56.484 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:56.492 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:48:56.563 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:56.563+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:56.609 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:56.609 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:56.609 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:56.609 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:56.609 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:56.617 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:56.642 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:56.642+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:56.743 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:56.743+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:56.761 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.502747644000003, 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.134442028000002, 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:48:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:48:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:48:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.442259773000004, 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:48:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:48:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:48:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:48:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.436402917000002, 2024-09-17T07:48:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:48:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:48:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.424709958000001, 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:48:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:48:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:48:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.355855374999997, 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:48:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:48:56.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.343956085999999, 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.800 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:48:56.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.293111539000002, 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.807 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:48:56.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:48:56.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:48:56.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.287103332000001, 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.813 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:48:56.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:56.819 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.281402974999999, 2024-09-17T07:48:56.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:48:56.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.820 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.820 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:48:56.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.275663350000002, 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.826 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:48:56.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:48:56.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:48:56.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 43.942552302000003, 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:48:56.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.097451405000001, 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:48:56.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.091364681999998, 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.846 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:48:56.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:48:56.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.007908381, 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:48:56.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:48:56.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:48:56.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.001593948999997, 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:56.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:48:56.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.864 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.995222489, 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.989300012999998, 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.931317116999999, 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 36.973612445000001, 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:48:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:48:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:48:56.887 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:56.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:56.890 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:56.888+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:56.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-17T07:48:56.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:48:57.002 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:57.002 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:57.002 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:48:57.002 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:57.002 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:57.002+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:57.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:57.059 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:57.075 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:57.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:57.090 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:57.090+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:57.091 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:57.091 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:57.092 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:57.108 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:57.109 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:57.109 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:48:57.111 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.111 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:57.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:57.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:57.191 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:57.190+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:57.218 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.218 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:57.292 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:57.291+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:57.328 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.329 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:57.393 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:57.393+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:57.438 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.438 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:57.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:48:57.487+0000 7f7c01ed4200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-17T07:48:57.494 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:57.493+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:57.589 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.589 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.595 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:57.595+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:57.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:57.696 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:57.695+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:57.699 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.699 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:57.797 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:57.796+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:57.809 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.809 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:57.901 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:57.901+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:57.959 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:57.959 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:57.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:58.009 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.009+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.067 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.067 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:58.100 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.100+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.173 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.174 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:58.201 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.201+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.281 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.281 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:58.302 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:58.301+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:58.397 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.397 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.403 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.403+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.405 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:58.504 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.504+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:48:58.549 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.549 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:58.605 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:58.605+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:58.706 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.706 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:58.706+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:58.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:48:58.808 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.807+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.816 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.816 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:48:58.909 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:58.909+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:58.967 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:58.968 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:58.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:48:59.011 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:59.010+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:59.112 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:59.110+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:59.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:48:59.118 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:59.118 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:59.127 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:48:59.212 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:59.212+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:48:59.225 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:48:59.225 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:48:59.232 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:48:59.232 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:48:59.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:48:59.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-09-17T07:48:59.239+0000 7f7c01ed4200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-17T07:48:59.247 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:48:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:48:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:48:59.263 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:48:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:48:59.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:48:59.313 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:59.314+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:59.332 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:59.332 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:48:59.332 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:48:59.332 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:48:59.332 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:59.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:48:59.415 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:59.415+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:59.480 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.313483730000002, 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:48:59.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.306144396000001, 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:48:59.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:48:59.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.305427322, 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:48:59.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:48:59.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.296199760999997, 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:48:59.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:48:59.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.286975193000004, 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.500 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.271500553000003, 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:59.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.262211043000001, 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:48:59.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:48:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.255620798000002, 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:48:59.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.239438317000001, 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:48:59.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:48:59.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.522 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.230363396999998, 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:48:59.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:48:59.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:48:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:48:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:48:59.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 51.889924520999998, 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:59.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:48:59.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 51.887872244, 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:48:59.535 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:48:59.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:48:59.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:59.541 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 51.827640592999998, 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:48:59.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:48:59.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.666461443999999, 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.548 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:48:59.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.666437143000003, 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.553 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:48:59.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:48:59.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.666402767000001, 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:48:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:48:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.563 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.66638141, 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:48:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:48:59.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.663712766000003, 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:48:59.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:48:59.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.653995193, 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:48:59.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:48:59.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:48:59.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 39.687884803999999, 2024-09-17T07:48:59.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:48:59.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:48:59.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:48:59.585 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:48:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:48:59.587 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:59.586+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:59.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:48:59.687 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:48:59.686+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:48:59.732 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:59.732 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:59.733 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:48:59.733 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:59.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:4.7 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:4.2 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:4.1 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:4.0 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:4.5 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:3.2 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:4.3 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:1.7 2024-09-17T07:48:59.768 INFO:teuthology.orchestra.run.smithi094.stdout:1.4 2024-09-17T07:48:59.769 INFO:teuthology.orchestra.run.smithi094.stdout:1.3 2024-09-17T07:48:59.769 INFO:teuthology.orchestra.run.smithi094.stdout:1.1 2024-09-17T07:48:59.788 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:48:59.787+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:48:59.843 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:48:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:48:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:48:59.843 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:48:59.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:48:59.893 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:59.893+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:59.990 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:48:59.989+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:48:59.995 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:59.996 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.828998071000001, 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:59.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:48:59.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:48:59.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.821338554999997, 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:00.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:00.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.812948986999999, 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:00.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:00.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.812577736999998, 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:00.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:00.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:00.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:00.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.804514771999997, 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:00.022 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:00.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:00.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.804143660000001, 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.794863255000003, 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.793203822000002, 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.787178996999998, 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.786928670000002, 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.779814832, 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.779297733, 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.760326566000003, 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.754069706000003, 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.747533644000001, 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.374667234, 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.372706827000002, 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.487038191000003, 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.176192553, 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.174831107999999, 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.112 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:00.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:00.114 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:00.112+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:00.213 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:00.213+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:00.256 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:00.256 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:00.256 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:00.256 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:00.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:00.285 INFO:tasks.thrashosds.thrasher:Moving pg 1.6 from osd.3 to osd.5 2024-09-17T07:49:00.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.6.3 2024-09-17T07:49:00.326 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:00.326+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:00.406 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:00.406 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:00.406 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:00.407 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:00.407 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:00.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:00.416 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:00.414+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:00.515 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:00.514+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:00.559 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:00.559 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:00.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:00.559 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:00.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.300396425999999, 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:00.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:00.564 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.932090809999998, 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:00.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.569 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.239908555, 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:00.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:00.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.575 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.234051698999998, 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:00.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:00.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.222358739999997, 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:00.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:00.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:00.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.153504157, 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.588 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:00.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:00.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.141604868000002, 2024-09-17T07:49:00.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.600 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.090760320999998, 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:00.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:00.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.084752113999997, 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.607 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:00.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.079051757000002, 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:00.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:00.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.073312131999998, 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:00.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:00.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 47.740201083999999, 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:00.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.895100186999997, 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:00.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:00.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.889013464000001, 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:00.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:00.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:00.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.805557163000003, 2024-09-17T07:49:00.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:00.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:00.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:00.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.799242731, 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:00.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:00.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:00.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.792871271000003, 2024-09-17T07:49:00.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:00.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:00.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.786948795000001, 2024-09-17T07:49:00.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:00.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:00.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.728965899000002, 2024-09-17T07:49:00.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:00.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:00.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:00.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.771261226999997, 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:00.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:00.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:00.678 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:00.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:00.680 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:00.679+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:00.781 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:00.779+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:00.785 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:00.785 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:00.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:00.882 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:00.880+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:00.932 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:00.932 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:00.932 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:00.932 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:00.932 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:00.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:00.982 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:00.982+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:01.078 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.911935765999999, 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.079 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:01.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:01.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:01.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:01.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:01.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:01.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:01.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.086 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.086 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:01.086 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.904596431999998, 2024-09-17T07:49:01.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:01.086 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.086 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.087 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:01.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.087 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:01.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:01.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:01.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:01.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:01.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:01.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:01.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:01.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:01.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:01.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:01.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:01.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:01.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:01.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:01.095 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.096 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.096 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.096 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.097 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:01.097 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.903879357999998, 2024-09-17T07:49:01.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:01.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.097 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.097 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:01.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:01.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:01.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:01.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:01.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:01.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:01.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:01.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:01.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:01.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:01.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:01.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:01.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:01.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:01.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:01.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:01.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:01.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:01.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:01.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.108 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.108 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.108 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:01.108 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.894651797000002, 2024-09-17T07:49:01.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:01.109 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.109 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:01.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.885427229000001, 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:01.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:01.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.869952589, 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:01.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.860663078999998, 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.854072834, 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:01.128 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:01.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.837890352999999, 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:01.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.828815433000003, 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:01.138 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:01.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:01.142 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.488376557000002, 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.143 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:01.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:01.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.486324279999998, 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:01.150 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:01.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:01.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.426092629000003, 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:01.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:01.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.264913479999997, 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:01.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:01.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:01.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.264889179000001, 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:01.167 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:01.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.264854802999999, 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.264833445999997, 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.262164802000001, 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.252447228999998, 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.286336839999997, 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:01.201 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:01.202 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:01.202+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:01.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:01.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:01.302 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:01.302+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:01.317 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:01.317 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:01.317 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:01.317 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:01.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:01.376 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:01.392 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:01.392 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:01.404 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:01.402+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:01.408 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:01.408 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:01.409 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:01.416 INFO:teuthology.orchestra.run.smithi094.stdout:get_pg_num_history pg_num_history pg_num_history(e68 pg_nums {1={12=8},2={12=1},3={16=4},4={46=8}} deleted_pools ) 2024-09-17T07:49:01.416 INFO:teuthology.orchestra.run.smithi094.stdout:Importing pgid 1.6 2024-09-17T07:49:01.416 INFO:teuthology.orchestra.run.smithi094.stdout:write_pg epoch 30 info 1.6( empty local-lis/les=29/30 n=0 ec=12/12 lis/c=29/29 les/c/f=30/30/0 sis=29) 2024-09-17T07:49:01.420 INFO:teuthology.orchestra.run.smithi094.stdout:Import successful 2024-09-17T07:49:01.424 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:01.428 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:01.431 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:01.432 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:01.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:01.505 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:01.504+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:01.583 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:01.583 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:01.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:49:01.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:01.633 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:01.632+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:01.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:01.707+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:01.736 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:01.736 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:01.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:01.809 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:01.808+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:01.847 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:01.847 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:01.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:01.909 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:01.908+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:01.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /var/log/ceph/exp.1.6.3 2024-09-17T07:49:01.997 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:01.997 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.003 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:49:02.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:02.044 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:02.044+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:02.110 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:02.109+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:02.145 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.145 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:02.211 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:02.211+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:02.273 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:02.273 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:48:59.514166+0000","last_up_change":"2024-09-17T07:48:52.499074+0000","last_in_change":"2024-09-17T07:46:43.251109+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":6,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":6,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"new_default_pool27356","create_time":"2024-09-17T07:48:02.539579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":29,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":31884},{"type":"v1","addr":"172.21.15.26:6801","nonce":31884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":31884},{"type":"v1","addr":"172.21.15.26:6803","nonce":31884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":31884},{"type":"v1","addr":"172.21.15.26:6807","nonce":31884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":31884},{"type":"v1","addr":"172.21.15.26:6805","nonce":31884}]},"public_addr":"172.21.15.26:6801/31884","cluster_addr":"172.21.15.26:6803/31884","heartbeat_back_addr":"172.21.15.26:6807/31884","heartbeat_front_addr":"172.21.15.26:6805/31884","state":["exists"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":0.8853302001953125,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":0,"in":1,"weight":0.3944549560546875,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":31604},{"type":"v1","addr":"172.21.15.94:6801","nonce":31604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":31604},{"type":"v1","addr":"172.21.15.94:6803","nonce":31604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":31604},{"type":"v1","addr":"172.21.15.94:6807","nonce":31604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":31604},{"type":"v1","addr":"172.21.15.94:6805","nonce":31604}]},"public_addr":"172.21.15.94:6801/31604","cluster_addr":"172.21.15.94:6803/31604","heartbeat_back_addr":"172.21.15.94:6807/31604","heartbeat_front_addr":"172.21.15.94:6805/31604","state":["exists"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":0.456390380859375,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":46,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":31606},{"type":"v1","addr":"172.21.15.94:6817","nonce":31606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":31606},{"type":"v1","addr":"172.21.15.94:6819","nonce":31606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":31606},{"type":"v1","addr":"172.21.15.94:6823","nonce":31606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":31606},{"type":"v1","addr":"172.21.15.94:6821","nonce":31606}]},"public_addr":"172.21.15.94:6817/31606","cluster_addr":"172.21.15.94:6819/31606","heartbeat_back_addr":"172.21.15.94:6823/31606","heartbeat_front_addr":"172.21.15.94:6821/31606","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":50},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":69}],"pg_upmap":[{"pgid":"1.5","osds":[1,0]}],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.2","osds":[2,4]},{"pgid":"3.1","osds":[2,4]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:49:02.286 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-17T07:49:02.286 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op apply-layout-settings --pool rbd 2024-09-17T07:49:02.290 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.290 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:02.327 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:02.327+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:02.412 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:02.412+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:02.436 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.436 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:02.513 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:02.513+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:02.525 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:49:02.550 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.550 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:02.614 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:02.614+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:02.700 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.701 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:02.716 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:02.715+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:02.811 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.811 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.816 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:02.816+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:02.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:02.917 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:02.916+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:02.962 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:02.962 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:02.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:03.018 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:03.018+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:03.072 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:03.072 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:03.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:03.124 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:03.124+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:03.181 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:03.181 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:03.189 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:03.233 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:03.233+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:03.289 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:03.289 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:03.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:03.320 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:03.320+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:03.386 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:49:03.421 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:03.421+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:03.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:03.436 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:03.436 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:03.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:03.522 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:03.522+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:03.532 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:03.548 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:03.564 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:03.564 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:03.564 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:03.579 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:03.580 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:03.580 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:03.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:03.589 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:03.589 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:03.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:03.590 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:03.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:03.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.419382210999999, 2024-09-17T07:49:03.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:03.591 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.591 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.591 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.591 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:03.595 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.412042876999998, 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.596 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:03.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:03.600 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.411325802999997, 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:03.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.402098242000001, 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:03.606 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:03.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.392873674000001, 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:03.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:03.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:03.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:03.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.377399034, 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:03.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.368109523999998, 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:03.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:03.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.361519278999999, 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:03.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.345336797999998, 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:03.635 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.336261878000002, 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:03.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:03.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.995823002000002, 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.640 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:03.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:03.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.993770724999997, 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:03.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:03.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:03.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.933539074000002, 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.653 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:03.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:03.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.659 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.772359925000003, 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:03.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.772335624, 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:03.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:03.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:03.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:03.671 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.772301247999998, 2024-09-17T07:49:03.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:03.671 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.671 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:03.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:03.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:03.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:03.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:03.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:03.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:03.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:03.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:03.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:03.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:03.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:03.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.680 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.680 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.772279890999997, 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:03.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:03.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:03.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:03.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:03.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:03.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:03.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.687 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.688 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:03.688 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:03.688 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.769611247, 2024-09-17T07:49:03.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:03.688 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.689 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:03.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:03.689 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:03.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.690 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:03.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:03.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:03.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:03.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:03.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:03.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:03.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:03.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:03.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:03.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:03.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:03.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:03.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:03.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:03.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:03.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:03.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:03.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:03.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.701 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:03.701 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:03.701 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.759893673999997, 2024-09-17T07:49:03.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:03.701 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:03.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.703 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:03.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:03.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:03.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:03.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:03.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:03.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:03.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:03.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:03.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:03.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:03.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:03.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:03.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:03.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:03.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 43.793783285000004, 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:03.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:03.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:03.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:03.718 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:03.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:03.720 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:03.719+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:03.821 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:03.820+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:03.824 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:03.825 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:03.832 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:03.903 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op apply-layout-settings --pool .mgr 2024-09-17T07:49:03.947 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:03.947+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:03.973 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:03.973 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:03.973 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:03.973 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:03.973 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:03.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:04.024 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:04.024+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.917347976000002, 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.909688459999998, 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:04.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.901298892, 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:04.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:04.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.900927641999999, 2024-09-17T07:49:04.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.892864676999999, 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:04.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.892493565000002, 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:04.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.883213159999997, 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:04.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.881553726999996, 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.875528901999999, 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.875278575000003, 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.868164737000001, 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.867647638000001, 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:04.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.848676470999997, 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:04.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:04.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.842419610999997, 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.149 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:04.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.835883549000002, 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:04.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:04.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.463017139000002, 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:04.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.461056732000003, 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 55.575388095999998, 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.264542458000001, 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.263181013000001, 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.188 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:04.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:04.190 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:04.190+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:04.290 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:04.290+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:04.334 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:04.334 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:04.334 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:04.334 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:04.342 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:04.426 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:04.426+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:04.484 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:04.484 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:04.484 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:04.484 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:04.484 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:04.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:04.493 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:04.492+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:04.538 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:49:04.620 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:04.620+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:04.641 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.380182668000003, 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.011877052000003, 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:04.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:04.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.319694796999997, 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:04.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:04.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:04.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:04.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:04.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.313837941000003, 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:04.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:04.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:04.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.666 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.302144982000002, 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:04.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:04.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:04.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:04.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:04.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:04.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:04.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:04.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:04.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:04.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.233290398999998, 2024-09-17T07:49:04.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:04.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.676 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.676 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.676 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.677 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:04.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.677 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:04.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:04.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:04.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.221391109999999, 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:04.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:04.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:04.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.170546563000002, 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:04.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:04.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:04.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.164538356000001, 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:04.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.158837998999999, 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:04.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.153098374000002, 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:04.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:04.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.819987326000003, 2024-09-17T07:49:04.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:04.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:04.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:04.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:04.726 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.974886429000001, 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:04.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:04.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:04.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:04.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:04.734 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.968799705999999, 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:04.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:04.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.885343405, 2024-09-17T07:49:04.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:04.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:04.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.879028972999997, 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.749 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:04.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.872657513, 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:04.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:04.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:04.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.866735036999998, 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:04.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.808752140999999, 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:04.770 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:04.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:04.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:04.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:04.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:04.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:04.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:04.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:04.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:04.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:04.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:04.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:04.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:04.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:04.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:04.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:04.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:04.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:04.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:04.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:04.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.780 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:04.780 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:04.781 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.851047469000001, 2024-09-17T07:49:04.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:04.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:04.781 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:04.781 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:04.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:04.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:04.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:04.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:04.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:04.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:04.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:04.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:04.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:04.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:04.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:04.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:04.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:04.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:04.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:04.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:04.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:04.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:04.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:04.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:04.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:04.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:04.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:04.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:04.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:04.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:04.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:04.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:04.790 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:04.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:04.794 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:04.791+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:04.891 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:04.892 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:04.892 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:04.892 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:04.893 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:04.893+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:04.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:04.994 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:49:04.994 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:04.994+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:04.999 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:04.999 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:05.000 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:05.000 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:05.000 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:05.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:05.095 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:05.095+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:05.151 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.984504805, 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.977165470999999, 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.976448396999999, 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.967220836000003, 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.957996268000002, 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.942521628000001, 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.933232117999999, 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.926641873000001, 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:05.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.910459392, 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:05.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.901384471999997, 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:05.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.560945596000003, 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.201 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:05.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.558893318999999, 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:05.207 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:05.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:05.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.498661667999997, 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:05.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:05.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:05.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:05.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.337482518999998, 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:05.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.337458218000002, 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:05.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.337423842, 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:05.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.337402484999998, 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:05.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.334733841000002, 2024-09-17T07:49:05.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.325016267999999, 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:05.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:05.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.358905878999998, 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:05.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:05.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:05.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:05.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:05.257 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:05.257 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:05.258 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:05.258+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:05.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:05.358 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:05.358+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:05.365 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:05.365 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:05.375 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:05.461 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:05.461+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:05.475 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:05.476 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:05.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:05.510 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op apply-layout-settings --pool cache 2024-09-17T07:49:05.533 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:49:05.562 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:05.561+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:05.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:05.591 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:05.591 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.424824815999997, 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:05.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:05.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.417165300000001, 2024-09-17T07:49:05.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:05.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:05.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.408775732000002, 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:05.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.408404482000002, 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:05.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.400341517000001, 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.399970404999998, 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:05.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.390689999999999, 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.389030566999999, 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.628 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:05.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.383005742000002, 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.382755414999998, 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.375641577000003, 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.375124477999996, 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.356153311, 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.349896450999999, 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.343360388999997, 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.970493978999997, 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.968533571999998, 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.082864936, 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:05.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:05.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.772019298000004, 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.770657853000003, 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:05.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:05.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:05.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:05.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:05.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:05.698 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:05.699 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:05.700 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:05.700 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:05.698+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:05.701 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:05.702 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:05.717 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:05.718 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:05.718 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:05.734 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:05.734 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:05.734 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:05.738 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:05.799 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:05.799+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:05.803 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:05.803 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:05.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:05.900 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:05.900+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:05.954 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:05.954 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:05.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:06.002 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:06.001+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:06.064 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.064 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:06.102 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:06.102+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:06.173 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.174 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:06.203 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:06.203+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:06.279 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.280 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:06.305 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:06.304+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:06.387 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.387 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:06.440 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:06.440+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:06.501 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.502 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.506 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:06.505+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:06.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:06.606 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:06.606+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:06.651 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.652 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:06.662 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:49:06.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:06.707+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:06.802 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.802 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.807 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:06.807+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:06.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:06.908 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:06.908+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:06.952 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:06.952 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:06.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:07.009 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:07.009+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:07.061 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:07.062 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:07.070 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:07.110 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:07.110+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:07.183 DEBUG:teuthology.orchestra.run.smithi094:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op apply-layout-settings --pool new_default_pool27356 2024-09-17T07:49:07.211 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:07.212 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:07.212 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:07.211+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:07.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:07.313 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:07.312+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:07.326 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:07.326 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:07.335 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:07.419 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:07.419+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:07.473 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:07.473 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:07.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:07.515 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:07.514+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:07.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:49:07.617 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:07.616+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:07.626 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:07.626 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:07.635 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:07.718 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:07.718+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:07.738 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:07.738 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:07.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:07.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:07.831 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:07.831+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:07.858 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:07.859 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:07.875 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:07.876 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:07.876 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:07.892 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:07.892 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:07.892 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:07.893 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:07.893 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:07.893 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:07.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:07.893 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:07.896 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:07.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:07.920 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:07.920+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:08.021 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:08.021+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:08.046 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:08.046 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:08.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:08.046 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.879642301000004, 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:08.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:08.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.871982785, 2024-09-17T07:49:08.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:08.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.863593217000002, 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.863221967000001, 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.855159001999994, 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.854787889999997, 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.845507484999999, 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:08.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.843848051999998, 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:08.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:08.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.087 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.837823227000001, 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.088 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:08.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.837572899999998, 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.830459062000003, 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.829941962999996, 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.810970796000007, 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:08.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.114 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.804713935999999, 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:08.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:08.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.798177874000004, 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:08.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:08.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:08.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.125 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.425311464000004, 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:08.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:08.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.423351056999998, 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:08.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:08.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:08.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 59.537682421, 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:08.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:08.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:08.144 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 55.226836783000003, 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:08.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:08.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:08.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 55.225475338000003, 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:08.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:08.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.155 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:08.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:08.157 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:08.156+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:08.257 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:08.257+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:08.300 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:08.300 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:08.301 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:08.301 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:08.304 INFO:teuthology.orchestra.run.smithi094.stderr:Nothing to do for non-filestore backend 2024-09-17T07:49:08.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:08.392 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:08.392+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:08.450 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:08.450 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:08.450 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:08.450 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:08.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:08.459 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:08.459+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:08.534 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:49:08.594 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:08.594+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:08.609 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 66.350616810000005, 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.982311193999998, 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.290128938999999, 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.284272082999998, 2024-09-17T07:49:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.272579124000004, 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:08.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.203724541, 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:08.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.191825252000001, 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.140980704999997, 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.134972498000003, 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.129272141000001, 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.123532515999997, 2024-09-17T07:49:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 55.790421467999998, 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:08.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.945320571000003, 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.939233848000001, 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.855777547000002, 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:08.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:08.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.849463114999999, 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:08.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.843091655000002, 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:08.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:08.711 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.837169179, 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.779186283000001, 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 48.821481611000003, 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:08.727 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:08.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:08.729 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:08.729+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:08.830 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:08.829+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:08.837 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:08.837 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:08.837 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:08.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:08.931 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:08.930+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:08.983 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:08.984 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:08.984 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:08.984 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:08.984 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:08.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:09.032 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:09.032+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:09.132 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:09.132 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:09.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.965216603000002, 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.133 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:09.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:09.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:09.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.957877268999994, 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:09.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:09.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:09.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.957160195, 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:09.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.947932633999997, 2024-09-17T07:49:09.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:09.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:09.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.938708066000004, 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.159 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:09.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:09.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:09.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.164 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.923233425999996, 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:09.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:09.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:09.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.913943915999994, 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.171 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:09.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:09.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:09.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.907353670999996, 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:09.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.891171189999994, 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:09.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:09.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:09.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.882096270000005, 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:09.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:09.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.541657393999998, 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:09.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:09.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:09.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:09.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.204 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.539605117000001, 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.479373465999998, 2024-09-17T07:49:09.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:09.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:09.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:09.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:09.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:09.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:09.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.318194317, 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.318170016000003, 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.318135640000001, 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.318114283, 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.315445639000004, 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 56.305728066, 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.339617677, 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:09.259 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:09.260 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:09.260+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:09.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:09.361 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:09.361+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:09.372 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:09.372 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:09.372 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:09.372 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:09.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:09.462 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:09.462+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:09.524 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:09.524 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:09.525 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:09.525 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:09.525 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:09.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:09.576 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:09.576+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.469867207999997, 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.637 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:09.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.462207692000007, 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.642 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:09.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.453818123999994, 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:09.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.453446873999994, 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:09.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.445383909, 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:09.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.662 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.445012797000004, 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:09.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.667 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.435732392000006, 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:09.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:09.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:09.672 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.434072959000005, 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:09.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:09.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.428048133999994, 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.677 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:09.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:09.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:09.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.427797807000005, 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:09.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.420683968999995, 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:09.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.420166870000003, 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:09.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:09.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:09.697 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.401195702999999, 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:09.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.394938843000006, 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:09.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:09.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.388402780999996, 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:09.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.015536371000003, 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:09.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.718 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.013575963999997, 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.719 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:09.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.127907327999999, 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.725 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:09.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.817061690000003, 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.815700245000002, 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:09.742 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:09.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:09.744 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:09.743+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:09.844 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:09.844+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:09.885 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:09.886 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:09.886 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:09.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:09.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:09.945 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:09.945+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:10.038 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:10.038 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:10.046 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.045+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:10.053 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:10.053 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:10.069 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:10.069 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:10.069 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:10.085 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:10.086 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:10.086 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:10.090 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:10.146 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.146+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.155 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.155 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:10.248 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.248+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.270 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.270 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:10.349 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.349+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.377 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.377 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:10.450 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.450+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.482 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.482 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:10.551 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:10.551+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:10.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:49:10.601 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.601 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:10.654 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.654+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.736 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.736 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:10.753 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:10.753+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:10.854 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.853+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:10.887 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:10.887 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:10.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:10.980 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:10.980+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:11.039 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.039 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:11.089 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:11.089+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:11.156 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:11.156+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:11.190 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.190 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:11.257 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:11.257+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:11.301 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.301 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:11.358 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:11.358+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:11.411 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.411 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.419 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:11.463 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:11.463+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:11.517 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.517 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:49:11.600 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:11.599+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:11.661 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:11.661+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:11.662 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.662 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:11.762 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:11.761+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:11.862 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:11.862+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:11.884 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:11.884 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:11.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:11.977 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:11.977+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:12.035 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:12.035 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:12.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:12.085 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:12.084+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:12.091 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:12.171 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:12.171+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:12.235 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:12.235 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:12.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:12.244 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:12.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:12.260 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:12.266 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:12.266+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:12.276 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:12.276 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:12.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:12.345 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:12.346 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:12.346 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:12.346 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:12.346 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:12.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:12.398 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:12.398+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:12.457 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 70.198288168000005, 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:12.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:12.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.829982552000004, 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:12.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.137800296999998, 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.473 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.131943441000004, 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:12.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.120250482000003, 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.051395898999999, 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.03949661, 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.988652063000004, 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.982643856000003, 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.976943499000001, 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:12.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.971203873999997, 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.516 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:12.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:12.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 59.638092825999998, 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.792991929000003, 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.786905206, 2024-09-17T07:49:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.703448905000002, 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.697134472999998, 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.690763013000002, 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.684840536999999, 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 56.626857641000001, 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.669152969000002, 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:12.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:12.576 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:12.575+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:12.676 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:12.676+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:12.683 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:12.683 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:12.683 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:12.684 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:12.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:12.777 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:12.777+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:12.831 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:12.831 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:12.831 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:12.831 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:12.831 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:12.839 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:12.880 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:12.880+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:12.979 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:12.979 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:12.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.812341586000002, 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:12.980 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:12.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:12.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.805002251999994, 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:12.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:12.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.804285178000001, 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:12.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:12.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:12.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:12.995 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.795057616999998, 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:12.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:12.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:12.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.000 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.785833049000004, 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.001 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.005 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.770358408999996, 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:13.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:13.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:13.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.761068898999994, 2024-09-17T07:49:13.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:13.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:13.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.754478653999996, 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.015 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:13.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.020 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.738296172999995, 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.025 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.729221253000006, 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:13.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:13.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:13.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:13.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:13.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:13.034 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.388782376999998, 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:13.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:13.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:13.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:13.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:13.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:13.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.386730099999994, 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:13.043 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:13.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.326498448999999, 2024-09-17T07:49:13.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:13.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.1653193, 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:13.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:13.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:13.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:13.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:13.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.062 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.165294998999997, 2024-09-17T07:49:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.063 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:13.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.064 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:13.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:13.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:13.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.165260623000002, 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:13.070 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:13.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.076 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.165239266, 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:13.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.162570621999997, 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.083 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 60.152853049000001, 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:13.088 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:13.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.18674266, 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:13.104 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:13.106 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:12.978+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.107 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:13.106+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:13.207 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:13.207+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.216 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:13.216 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:13.216 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:13.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:13.308 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:13.308+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.367 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:13.367 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:13.367 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:13.367 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:13.367 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:13.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:13.408 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:13.408+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:13.509 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:13.509+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.352532939, 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.344873422999996, 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:13.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.530 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.336483854999997, 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:13.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:13.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.336112604999997, 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:13.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.328049640000003, 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:13.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:13.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.327678528000007, 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.318398122999994, 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.316738689999994, 2024-09-17T07:49:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.310713864999997, 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.310463537999993, 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.303349699999998, 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.302832601000006, 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.283861434000002, 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.277604573999994, 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.271068511999999, 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.898202101999999, 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.896241695000001, 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.010573058999995, 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.699727420999999, 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 60.698365975999998, 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:13.625 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:13.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:49:13.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:13.627 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:13.626+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.727 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:13.727+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:13.772 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:13.772 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:13.772 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:13.772 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:13.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:13.828 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:13.828+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:13.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:13.831 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:13.831 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:49:13.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:49:13.884 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:13.884 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:13.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:13.937 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:13.937+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.754831108999994, 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 66.386525492999994, 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.694343238000002, 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.688486381999994, 2024-09-17T07:49:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.676793423000007, 2024-09-17T07:49:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.607938840000003, 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.596039551000004, 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:14.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.545195003999993, 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.539186796999999, 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.533486440000004, 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.527746815, 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.194635767000001, 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.349534869999999, 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.343448146999997, 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.259991845999998, 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.253677414000002, 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.247305953999998, 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.241383478000003, 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.183400581999997, 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.225695909999999, 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.152 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:14.154 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 76, 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:49:11.540307+0000", 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:48:52.499074+0000", 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:46:43.251109+0000", 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:49:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5801984, 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 4, 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:49:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:49:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:49:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "new_default_pool27356", 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:48:02.539579+0000", 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "49", 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 49, 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:49:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 29, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/31884", 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/31884", 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/31884", 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/31884", 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 69, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:49:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 69, 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:49:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.3944549560546875, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:49:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/31604", 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/31604", 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/31604", 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/31604", 2024-09-17T07:49:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.456390380859375, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:49:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:49:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 46, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:49:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/31606", 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/31606", 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/31606", 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/31606", 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.2", 2024-09-17T07:49:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:49:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [ 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "snaps": [ 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "begin": 2, 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "length": 1 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:14.235 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:14.236 INFO:tasks.thrashosds.thrasher:j is {'epoch': 76, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:49:11.540307+0000', 'last_up_change': '2024-09-17T07:48:52.499074+0000', 'last_in_change': '2024-09-17T07:46:43.251109+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 29, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 31884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 31884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 31884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 31884}]}, 'public_addr': '172.21.15.26:6801/31884', 'cluster_addr': '172.21.15.26:6803/31884', 'heartbeat_back_addr': '172.21.15.26:6807/31884', 'heartbeat_front_addr': '172.21.15.26:6805/31884', 'state': ['exists']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 0, 'in': 1, 'weight': 0.3944549560546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 31604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 31604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 31604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 31604}]}, 'public_addr': '172.21.15.94:6801/31604', 'cluster_addr': '172.21.15.94:6803/31604', 'heartbeat_back_addr': '172.21.15.94:6807/31604', 'heartbeat_front_addr': '172.21.15.94:6805/31604', 'state': ['exists']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 0.456390380859375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 31606}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 31606}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 31606}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 31606}]}, 'public_addr': '172.21.15.94:6817/31606', 'cluster_addr': '172.21.15.94:6819/31606', 'heartbeat_back_addr': '172.21.15.94:6823/31606', 'heartbeat_front_addr': '172.21.15.94:6821/31606', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '1.2', 'osds': [2, 4]}, {'pgid': '3.1', 'osds': [2, 4]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:49:14.237 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:49:14.237 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:14.153+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:14.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:14.281 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:14.281+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:14.347 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:14.347 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:14.347 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:14.347 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:14.354 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:14.354+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:14.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:14.387 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:14.402 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:14.402 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:14.418 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:14.434 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:14.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:14.455 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:14.455+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:14.495 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:14.495 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:14.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:14.517 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:49:14.528 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-17T07:49:13.436555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215549224,"num_objects":96,"num_object_clones":1,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":6,"num_read":3892,"num_read_kb":175712,"num_write":681,"num_write_kb":264994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165881,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":316,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":691,"ondisk_log_size":691,"up":23,"acting":25,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2147596,"kb_used_data":334328,"kb_used_omap":9,"kb_used_meta":1813174,"kb_avail":564083444,"statfs":{"total":579820584960,"available":577621446656,"internally_reserved":0,"allocated":342351872,"data_stored":342711788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1856690192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.904424"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":26,"reported_epoch":67,"state":"stale+active+undersized","last_fresh":"2024-09-17T07:48:49.979339+0000","last_change":"2024-09-17T07:48:03.640213+0000","last_active":"2024-09-17T07:48:49.979339+0000","last_peered":"2024-09-17T07:48:49.979339+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640213+0000","last_became_peered":"2024-09-17T07:48:03.640213+0000","last_unstale":"2024-09-17T07:48:52.503858+0000","last_undegraded":"2024-09-17T07:48:49.979339+0000","last_fullsized":"2024-09-17T07:48:03.639399+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:21.591427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"45'7","reported_seq":80,"reported_epoch":48,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:05.372036+0000","last_change":"2024-09-17T07:48:01.879165+0000","last_active":"2024-09-17T07:48:05.372036+0000","last_peered":"2024-09-17T07:48:05.372036+0000","last_clean":"2024-09-17T07:47:41.593387+0000","last_became_active":"2024-09-17T07:47:45.602779+0000","last_became_peered":"2024-09-17T07:47:45.602779+0000","last_unstale":"2024-09-17T07:48:08.649038+0000","last_undegraded":"2024-09-17T07:48:01.878970+0000","last_fullsized":"2024-09-17T07:47:45.602289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:18.255015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":871,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.534087+0000","last_change":"2024-09-17T07:48:53.510147+0000","last_active":"2024-09-17T07:49:08.534087+0000","last_peered":"2024-09-17T07:49:08.534087+0000","last_clean":"2024-09-17T07:48:50.494421+0000","last_became_active":"2024-09-17T07:48:53.510147+0000","last_became_peered":"2024-09-17T07:48:53.510147+0000","last_unstale":"2024-09-17T07:49:08.534087+0000","last_undegraded":"2024-09-17T07:48:53.509603+0000","last_fullsized":"2024-09-17T07:48:53.509541+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:52:15.656871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":756,"num_read_kb":27413,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":43,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-17T07:49:08.534078+0000","last_change":"2024-09-17T07:48:05.649758+0000","last_active":"2024-09-17T07:49:08.534078+0000","last_peered":"2024-09-17T07:49:08.534078+0000","last_clean":"2024-09-17T07:49:08.534078+0000","last_became_active":"2024-09-17T07:48:03.642170+0000","last_became_peered":"2024-09-17T07:48:03.642170+0000","last_unstale":"2024-09-17T07:49:08.534078+0000","last_undegraded":"2024-09-17T07:49:08.534078+0000","last_fullsized":"2024-09-17T07:49:08.534078+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:30.026257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"54'4","reported_seq":79,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979236+0000","last_change":"2024-09-17T07:48:02.160661+0000","last_active":"2024-09-17T07:48:49.979236+0000","last_peered":"2024-09-17T07:48:49.979236+0000","last_clean":"2024-09-17T07:47:42.119185+0000","last_became_active":"2024-09-17T07:47:45.604354+0000","last_became_peered":"2024-09-17T07:47:45.604354+0000","last_unstale":"2024-09-17T07:48:52.503859+0000","last_undegraded":"2024-09-17T07:48:02.140633+0000","last_fullsized":"2024-09-17T07:47:45.603702+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:54:04.231757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"58'257","reported_seq":1183,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.533912+0000","last_change":"2024-09-17T07:48:01.916903+0000","last_active":"2024-09-17T07:49:08.533912+0000","last_peered":"2024-09-17T07:49:08.533912+0000","last_clean":"2024-09-17T07:47:43.970152+0000","last_became_active":"2024-09-17T07:47:45.604092+0000","last_became_peered":"2024-09-17T07:47:45.604092+0000","last_unstale":"2024-09-17T07:49:08.533912+0000","last_undegraded":"2024-09-17T07:48:01.915803+0000","last_fullsized":"2024-09-17T07:47:45.603408+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:28.677664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797533,"num_objects":17,"num_object_clones":1,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1339,"num_read_kb":77312,"num_write":306,"num_write_kb":58504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":47,"reported_epoch":75,"state":"active+undersized","last_fresh":"2024-09-17T07:49:08.534276+0000","last_change":"2024-09-17T07:48:53.510166+0000","last_active":"2024-09-17T07:49:08.534276+0000","last_peered":"2024-09-17T07:49:08.534276+0000","last_clean":"2024-09-17T07:48:50.494734+0000","last_became_active":"2024-09-17T07:48:53.510166+0000","last_became_peered":"2024-09-17T07:48:53.510166+0000","last_unstale":"2024-09-17T07:49:08.534276+0000","last_undegraded":"2024-09-17T07:49:08.534276+0000","last_fullsized":"2024-09-17T07:48:53.509462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:33.475636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"45'5","reported_seq":53,"reported_epoch":75,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:08.533760+0000","last_change":"2024-09-17T07:48:04.736433+0000","last_active":"2024-09-17T07:49:08.533760+0000","last_peered":"2024-09-17T07:49:08.533760+0000","last_clean":"2024-09-17T07:49:08.533760+0000","last_became_active":"2024-09-17T07:48:04.643512+0000","last_became_peered":"2024-09-17T07:48:04.643512+0000","last_unstale":"2024-09-17T07:49:08.533760+0000","last_undegraded":"2024-09-17T07:49:08.533760+0000","last_fullsized":"2024-09-17T07:49:08.533760+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:58.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"71'184","reported_seq":922,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-17T07:49:11.545396+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:49:11.545396+0000","last_peered":"2024-09-17T07:49:11.545396+0000","last_clean":"2024-09-17T07:49:11.545396+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:49:11.545396+0000","last_undegraded":"2024-09-17T07:49:11.545396+0000","last_fullsized":"2024-09-17T07:49:11.545396+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:04:40.147948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777322,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1408,"num_read_kb":46186,"num_write":100,"num_write_kb":32777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":106,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+undersized","last_fresh":"2024-09-17T07:49:08.534231+0000","last_change":"2024-09-17T07:48:03.641341+0000","last_active":"2024-09-17T07:49:08.534231+0000","last_peered":"2024-09-17T07:49:08.534231+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.641341+0000","last_became_peered":"2024-09-17T07:48:03.641341+0000","last_unstale":"2024-09-17T07:49:08.534231+0000","last_undegraded":"2024-09-17T07:49:08.534231+0000","last_fullsized":"2024-09-17T07:48:03.640567+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:02.148375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":112,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.534072+0000","last_change":"2024-09-17T07:48:53.509681+0000","last_active":"2024-09-17T07:49:08.534072+0000","last_peered":"2024-09-17T07:49:08.534072+0000","last_clean":"2024-09-17T07:48:50.494470+0000","last_became_active":"2024-09-17T07:48:53.509681+0000","last_became_peered":"2024-09-17T07:48:53.509681+0000","last_unstale":"2024-09-17T07:49:08.534072+0000","last_undegraded":"2024-09-17T07:48:53.508978+0000","last_fullsized":"2024-09-17T07:48:53.508899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:56.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":130,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.533992+0000","last_change":"2024-09-17T07:48:09.653077+0000","last_active":"2024-09-17T07:49:08.533992+0000","last_peered":"2024-09-17T07:49:08.533992+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:48:09.653077+0000","last_became_peered":"2024-09-17T07:48:09.653077+0000","last_unstale":"2024-09-17T07:49:08.533992+0000","last_undegraded":"2024-09-17T07:48:09.652573+0000","last_fullsized":"2024-09-17T07:48:09.652495+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:35.420073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"55'78","reported_seq":213,"reported_epoch":75,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:08.533674+0000","last_change":"2024-09-17T07:48:04.691716+0000","last_active":"2024-09-17T07:49:08.533674+0000","last_peered":"2024-09-17T07:49:08.533674+0000","last_clean":"2024-09-17T07:49:08.533674+0000","last_became_active":"2024-09-17T07:48:04.643230+0000","last_became_peered":"2024-09-17T07:48:04.643230+0000","last_unstale":"2024-09-17T07:49:08.533674+0000","last_undegraded":"2024-09-17T07:49:08.533674+0000","last_fullsized":"2024-09-17T07:49:08.533674+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:16:12.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":88,"num_write_kb":24609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":27,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979315+0000","last_change":"2024-09-17T07:48:26.474017+0000","last_active":"2024-09-17T07:48:49.979315+0000","last_peered":"2024-09-17T07:48:49.979315+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640668+0000","last_became_peered":"2024-09-17T07:48:03.640668+0000","last_unstale":"2024-09-17T07:48:52.503860+0000","last_undegraded":"2024-09-17T07:48:25.341555+0000","last_fullsized":"2024-09-17T07:48:03.639432+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:44.698558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":104,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.534159+0000","last_change":"2024-09-17T07:48:53.509922+0000","last_active":"2024-09-17T07:49:08.534159+0000","last_peered":"2024-09-17T07:49:08.534159+0000","last_clean":"2024-09-17T07:48:50.494622+0000","last_became_active":"2024-09-17T07:48:53.509922+0000","last_became_peered":"2024-09-17T07:48:53.509922+0000","last_unstale":"2024-09-17T07:49:08.534159+0000","last_undegraded":"2024-09-17T07:48:53.509331+0000","last_fullsized":"2024-09-17T07:48:53.509273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:23.503950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":38,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979198+0000","last_change":"2024-09-17T07:48:25.422858+0000","last_active":"2024-09-17T07:48:49.979198+0000","last_peered":"2024-09-17T07:48:49.979198+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640271+0000","last_became_peered":"2024-09-17T07:48:03.640271+0000","last_unstale":"2024-09-17T07:48:52.503861+0000","last_undegraded":"2024-09-17T07:48:25.340395+0000","last_fullsized":"2024-09-17T07:48:03.639357+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:03.926215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":65,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.534327+0000","last_change":"2024-09-17T07:48:01.981238+0000","last_active":"2024-09-17T07:49:08.534327+0000","last_peered":"2024-09-17T07:49:08.534327+0000","last_clean":"2024-09-17T07:47:41.593808+0000","last_became_active":"2024-09-17T07:47:45.603610+0000","last_became_peered":"2024-09-17T07:47:45.603610+0000","last_unstale":"2024-09-17T07:49:08.534327+0000","last_undegraded":"2024-09-17T07:48:01.980889+0000","last_fullsized":"2024-09-17T07:47:45.603119+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:18:17.830983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":30,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979225+0000","last_change":"2024-09-17T07:48:04.643889+0000","last_active":"2024-09-17T07:48:49.979225+0000","last_peered":"2024-09-17T07:48:49.979225+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640398+0000","last_became_peered":"2024-09-17T07:48:03.640398+0000","last_unstale":"2024-09-17T07:48:52.503862+0000","last_undegraded":"2024-09-17T07:48:03.642853+0000","last_fullsized":"2024-09-17T07:48:03.639625+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:08.008675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":57,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.533903+0000","last_change":"2024-09-17T07:48:01.880716+0000","last_active":"2024-09-17T07:49:08.533903+0000","last_peered":"2024-09-17T07:49:08.533903+0000","last_clean":"2024-09-17T07:46:51.988421+0000","last_became_active":"2024-09-17T07:47:45.604099+0000","last_became_peered":"2024-09-17T07:47:45.604099+0000","last_unstale":"2024-09-17T07:49:08.533903+0000","last_undegraded":"2024-09-17T07:48:01.880450+0000","last_fullsized":"2024-09-17T07:47:45.603421+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:28.433406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":53,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.533922+0000","last_change":"2024-09-17T07:48:53.509737+0000","last_active":"2024-09-17T07:49:08.533922+0000","last_peered":"2024-09-17T07:49:08.533922+0000","last_clean":"2024-09-17T07:48:50.494367+0000","last_became_active":"2024-09-17T07:48:53.509737+0000","last_became_peered":"2024-09-17T07:48:53.509737+0000","last_unstale":"2024-09-17T07:49:08.533922+0000","last_undegraded":"2024-09-17T07:48:53.508926+0000","last_fullsized":"2024-09-17T07:48:53.508829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"45'5","reported_seq":64,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:08.533844+0000","last_change":"2024-09-17T07:48:53.508703+0000","last_active":"2024-09-17T07:49:08.533844+0000","last_peered":"2024-09-17T07:49:08.533844+0000","last_clean":"2024-09-17T07:48:02.144100+0000","last_became_active":"2024-09-17T07:48:53.508703+0000","last_became_peered":"2024-09-17T07:48:53.508703+0000","last_unstale":"2024-09-17T07:49:08.533844+0000","last_undegraded":"2024-09-17T07:48:53.508098+0000","last_fullsized":"2024-09-17T07:48:53.507992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:19.938198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":89,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":3},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80740871,"num_objects":43,"num_object_clones":1,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3717,"num_read_kb":175577,"num_write":530,"num_write_kb":132276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":316,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122744832,"data_stored":125829937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":564,"ondisk_log_size":564,"up":5,"acting":6,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":1,"acting":1,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217915,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":44,"num_write_kb":131084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213954560,"data_stored":213909821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":8,"acting":9,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640283,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388356,"kb_used_data":86836,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93983484,"statfs":{"total":96636764160,"available":96239087616,"internally_reserved":0,"allocated":88920064,"data_stored":88404740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57599999999999996}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54900000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.44},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.33700000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640287,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373080,"kb_used_data":71564,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93998760,"statfs":{"total":96636764160,"available":96254730240,"internally_reserved":0,"allocated":73281536,"data_stored":75865559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.625,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40300000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.254}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.316},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.32800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319040,"kb_used_data":17516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94052800,"statfs":{"total":96636764160,"available":96310067200,"internally_reserved":0,"allocated":17936384,"data_stored":17586425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.309}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.24299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355652,"kb_used_data":54136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94016188,"statfs":{"total":96636764160,"available":96272576512,"internally_reserved":0,"allocated":55435264,"data_stored":54888545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.315,"15min":0.32700000000000001},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.314,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.247,"5min":0.30299999999999999,"15min":0.312},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.35299999999999998,"5min":0.434,"15min":0.434},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.628,"15min":0.628},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.45500000000000002,"15min":0.46899999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.49399999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.44600000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438}]}]},{"osd":1,"up_from":11,"seq":47244640288,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409540,"kb_used_data":103864,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93962300,"statfs":{"total":96636764160,"available":96217395200,"internally_reserved":0,"allocated":106356736,"data_stored":105810585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.441,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.35499999999999998,"15min":0.36099999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.498,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.28599999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61299999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.58099999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.374}]}]},{"osd":0,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301928,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069912,"statfs":{"total":96636764160,"available":96327589888,"internally_reserved":0,"allocated":421888,"data_stored":155934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16777216,"data_stored":16777216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75517952,"data_stored":75497640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29372416,"data_stored":29360350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29372416,"data_stored":29360310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51408896,"data_stored":54526249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:49:14.530 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [4, 0, 5, 3] 2024-09-17T07:49:14.530 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '4', '0', '5', '3'] 2024-09-17T07:49:14.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.0 4 0 5 3 2024-09-17T07:49:14.547 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:49:14.556 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:14.555+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:14.641 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:14.641 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:14.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:14.693 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:14.693+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:14.752 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:14.752 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:14.757 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:14.757+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:14.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:14.858 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:14.858+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:14.903 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:14.903 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:14.911 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:14.995 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:14.995+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:15.052 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.053 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.060 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:15.059+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:15.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:15.160 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:15.160+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:15.203 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.203 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:15.261 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:15.261+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:15.312 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.313 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:15.362 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:15.362+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:15.422 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.422 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:15.463 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:15.463+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:15.538 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.538 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.547 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:15.553 INFO:teuthology.orchestra.run.smithi026.stderr:set 2.0 pg_upmap_items mapping to [4->0,5->3] 2024-09-17T07:49:15.564 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:15.564+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:15.644 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.644 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:15.665 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:15.665+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:15.752 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.752 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:15.805 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:15.805+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:15.866 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:15.865+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:15.866 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:15.867 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:15.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:15.967 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:15.966+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:16.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:16.067 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:16.067+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:16.168 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:16.168 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:16.168 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:16.168+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:16.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:16.269 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:16.268+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:16.277 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:16.277 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:16.286 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:16.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:16.370+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:16.386 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:16.387 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:16.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:16.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:16.482 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:16.482+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:16.498 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:16.498 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:16.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:16.540 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:16.556 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:16.556 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:16.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:49:16.572 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:16.572 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:16.572+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:16.588 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:16.592 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:16.610 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:16.610 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:16.610 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:16.611 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:16.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:16.672 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:16.672+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:16.717 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:16.717 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:16.718 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:16.718 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:16.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:16.773 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:16.772+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.658141248999996, 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.650801915000002, 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.650084840999995, 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.640857280000006, 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.631632711999998, 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.616158072000005, 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.606868562000002, 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.600278317000004, 2024-09-17T07:49:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.584095836000003, 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.575020916, 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:16.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.234582040000006, 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.232529763000002, 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:16.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:16.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.884 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.172298111999993, 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:16.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:16.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:16.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.011118963000001, 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.891 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:16.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.011094662000005, 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:16.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.011060286000003, 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:16.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:16.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.011038928999994, 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.907 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:16.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:16.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:16.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.008370284999998, 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:16.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:16.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.998652712000002, 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:16.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:16.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.032542323000001, 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:16.923 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:16.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:16.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:16.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:16.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.929 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:16.929 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:16.929 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:16.931 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:16.930+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:16.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:17.032 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:17.031+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:17.033 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:17.033 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:17.033 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:17.033 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:17.042 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:17.133 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:17.132+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:17.142 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:17.142 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:17.142 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:17.143 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:17.143 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:17.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:17.235 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:17.235+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.127017456999994, 2024-09-17T07:49:17.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:17.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:17.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.119357941000004, 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:17.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:17.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:17.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.110968373000006, 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:17.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.314 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.110597123000005, 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:17.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:17.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.102534157999997, 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.320 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:17.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:17.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.325 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.326 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.102163046000001, 2024-09-17T07:49:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.092882641000003, 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.091223208000002, 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.085198383000005, 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.084948056000002, 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.351 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.077834218000007, 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.077317119, 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:17.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:17.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.058345951999996, 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:17.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:17.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.052089092000003, 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.377 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:17.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.381 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.045553029999994, 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:17.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.388 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.672686619999993, 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:17.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:17.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:17.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.670726212999995, 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:17.396 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:17.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:17.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.403 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.785057577000003, 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.404 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:17.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.474211939, 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.472850493999999, 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.425 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:17.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:17.427 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:17.426+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:17.527 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:17.527+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:17.555 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:49:17.572 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:17.572 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:17.572 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:17.572 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:17.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:17.665 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:17.664+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:17.723 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:17.723 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:17.729 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:17.727+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:17.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:17.829 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:17.829+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:17.875 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.616239096000001, 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:17.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:17.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 70.24793348, 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:17.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:17.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.555751224999995, 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:17.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:17.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.549894369, 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:17.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.538201409999999, 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.896 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:17.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:17.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.902 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.469346826999995, 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:17.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:17.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:17.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.457447537999997, 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:17.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:17.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.406602991, 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.915 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:17.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:17.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:17.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.400594784000006, 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:17.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:17.927 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.394894426999997, 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:17.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.389154801999993, 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.056043754000001, 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:17.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.210942856999999, 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:17.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:17.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.204856134000003, 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:17.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:17.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.121399832999998, 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:17.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:17.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.115085401000002, 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.964 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:17.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.108713940999998, 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.970 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:17.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:17.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:17.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.102791465000003, 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:17.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:17.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.044808568999997, 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:17.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:17.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.087103896999999, 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:17.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:17.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:17.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:17.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:17.992 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:17.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:17.994 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:17.993+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:18.094 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:18.093+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:18.103 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:18.103 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:18.103 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:18.103 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:18.112 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:18.195 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:18.193+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:18.210 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:18.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:18.296 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:18.295+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:18.316 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:18.316 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:18.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:18.316 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:18.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.316 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.149718191999995, 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:18.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.142378858000001, 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:18.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.325 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.141661783999993, 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:18.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:18.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.132434223000004, 2024-09-17T07:49:18.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:18.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:18.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:18.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.123209654999997, 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:18.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:18.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:18.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.107735015000003, 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.339 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:18.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.098445505000001, 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.091855260000003, 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:18.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:18.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:18.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.075672779000001, 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.352 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:18.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.066597858999998, 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:18.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.726158983000005, 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:18.361 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:18.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:18.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:18.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:18.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.724106706000001, 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:18.368 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.373 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.663875055000005, 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:18.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.502695906, 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:18.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:18.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.502671605000003, 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:18.385 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:18.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.502637229000001, 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:18.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:18.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:18.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.502615872000007, 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:18.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:18.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.499947227999996, 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:18.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:18.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.490229654999993, 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:18.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.524119266, 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:18.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:18.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:18.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:18.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:18.417 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:18.419 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:18.418+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:18.419 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:18.520 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:18.520+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:18.528 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:18.528 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:18.529 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:18.529 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:18.537 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:18.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:18.637 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:18.636+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:18.640 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:18.641 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:18.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:18.695 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:18.722 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:18.722+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:18.794 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.627920423000006, 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.620260907000002, 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:18.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.611871339000004, 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.611500089000003, 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.603437123999996, 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.603066011999999, 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.593785607000001, 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.592126174000001, 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.586101349000003, 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.585851022, 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.578737184000005, 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.578220084999998, 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.559248917999994, 2024-09-17T07:49:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.552992058000001, 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.546455996000006, 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.173589586000006, 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:18.871 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:18.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.171629179000007, 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:18.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:18.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 70.285960543000002, 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.975114904999998, 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.973753459999998, 2024-09-17T07:49:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:18.901 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:18.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:18.903 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:18.901+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:18.927 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:18.944 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:18.945 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:18.945 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:18.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:18.948 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:19.004 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.002+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.014 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.014 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:19.106 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.105+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.164 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.164 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:19.215 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.213+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.306 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:19.304+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:19.315 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.315 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:19.407 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.406+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.433 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.433 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:19.508 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:19.507+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:19.540 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.540 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.549 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:19.570 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:49:19.614 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.613+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.646 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.646 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:19.713 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.709+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.759 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.759 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:19.851 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:19.850+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:19.909 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:19.909 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:19.911 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:19.911+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:19.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:20.012 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:20.011+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:20.061 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.061 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:20.153 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:20.152+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:20.211 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.212 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.214 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:20.212+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:20.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:20.314 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:20.314+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:20.362 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.362 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:20.415 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:20.415+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:20.472 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.472 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:20.516 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:20.515+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:20.564 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:49:20.567 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:20.567 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:20.568 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:49:20.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:49:20.592 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.593 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.602 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:20.619 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:20.618+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:20.702 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.702 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.710 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:20.718 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:20.716+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:20.808 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.808 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:20.818 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:20.818+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:20.840 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:20.840 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 80, 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:49:20.560979+0000", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:48:52.499074+0000", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:46:43.251109+0000", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:49:20.841 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 4, 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:49:20.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:49:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:49:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:49:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:49:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "new_default_pool27356", 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:48:02.539579+0000", 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "49", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 49, 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:20.863 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:20.864 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:20.865 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:20.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 29, 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:49:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/31884", 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/31884", 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/31884", 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/31884", 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:20.871 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 69, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:49:20.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:49:20.876 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:20.877 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 69, 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:49:20.882 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.3944549560546875, 2024-09-17T07:49:20.883 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:49:20.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/31604", 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/31604", 2024-09-17T07:49:20.888 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/31604", 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/31604", 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.456390380859375, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 46, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/31606", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/31606", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/31606", 2024-09-17T07:49:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/31606", 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:49:20.904 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.2", 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:49:20.906 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [ 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "snaps": [ 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "begin": 2, 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: "length": 1 2024-09-17T07:49:20.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:49:20.908 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:49:20.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:20.909 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:20.910 INFO:tasks.thrashosds.thrasher:j is {'epoch': 80, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:49:20.560979+0000', 'last_up_change': '2024-09-17T07:48:52.499074+0000', 'last_in_change': '2024-09-17T07:46:43.251109+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 29, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 31884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 31884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 31884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 31884}]}, 'public_addr': '172.21.15.26:6801/31884', 'cluster_addr': '172.21.15.26:6803/31884', 'heartbeat_back_addr': '172.21.15.26:6807/31884', 'heartbeat_front_addr': '172.21.15.26:6805/31884', 'state': ['exists']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 0, 'in': 1, 'weight': 0.3944549560546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 31604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 31604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 31604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 31604}]}, 'public_addr': '172.21.15.94:6801/31604', 'cluster_addr': '172.21.15.94:6803/31604', 'heartbeat_back_addr': '172.21.15.94:6807/31604', 'heartbeat_front_addr': '172.21.15.94:6805/31604', 'state': ['exists']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 0.456390380859375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 31606}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 31606}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 31606}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 31606}]}, 'public_addr': '172.21.15.94:6817/31606', 'cluster_addr': '172.21.15.94:6819/31606', 'heartbeat_back_addr': '172.21.15.94:6823/31606', 'heartbeat_front_addr': '172.21.15.94:6821/31606', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '1.2', 'osds': [2, 4]}, {'pgid': '3.1', 'osds': [2, 4]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:49:20.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:49:20.918 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:20.918+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:20.919 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:20.919 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:20.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:20.949 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:21.019 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:21.017+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:21.074 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:21.075 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:21.075 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:21.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:21.091 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:21.092 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:21.109 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:21.110 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:21.110 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:21.120 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:21.119+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:21.125 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:21.126 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:21.126 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:21.130 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.02930422, 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.021644703999996, 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.013255135999998, 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.012883885999997, 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.004820921000004, 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.004449808999993, 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.995169403999995, 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.993509970999995, 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.987485145999997, 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.987234818999994, 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.980120980999999, 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.979603882000006, 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.960632715000003, 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.954375854999995, 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.947839793, 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.574973383, 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.573012976000001, 2024-09-17T07:49:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.687344339999996, 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.376498702000006, 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.375137257000006, 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.299 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:21.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:21.302 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:21.302 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:49:21.303 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-17T07:49:20.562936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215549224,"num_objects":96,"num_object_clones":1,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":6,"num_read":3892,"num_read_kb":175712,"num_write":681,"num_write_kb":264994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165881,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":316,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":691,"ondisk_log_size":691,"up":23,"acting":25,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2147704,"kb_used_data":334424,"kb_used_omap":9,"kb_used_meta":1813174,"kb_avail":564083336,"statfs":{"total":579820584960,"available":577621336064,"internally_reserved":0,"allocated":342450176,"data_stored":342755303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1856690192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.017627"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":26,"reported_epoch":67,"state":"stale+active+undersized","last_fresh":"2024-09-17T07:48:49.979339+0000","last_change":"2024-09-17T07:48:03.640213+0000","last_active":"2024-09-17T07:48:49.979339+0000","last_peered":"2024-09-17T07:48:49.979339+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640213+0000","last_became_peered":"2024-09-17T07:48:03.640213+0000","last_unstale":"2024-09-17T07:48:52.503858+0000","last_undegraded":"2024-09-17T07:48:49.979339+0000","last_fullsized":"2024-09-17T07:48:03.639399+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:21.591427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"45'7","reported_seq":80,"reported_epoch":48,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:05.372036+0000","last_change":"2024-09-17T07:48:01.879165+0000","last_active":"2024-09-17T07:48:05.372036+0000","last_peered":"2024-09-17T07:48:05.372036+0000","last_clean":"2024-09-17T07:47:41.593387+0000","last_became_active":"2024-09-17T07:47:45.602779+0000","last_became_peered":"2024-09-17T07:47:45.602779+0000","last_unstale":"2024-09-17T07:48:08.649038+0000","last_undegraded":"2024-09-17T07:48:01.878970+0000","last_fullsized":"2024-09-17T07:47:45.602289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:18.255015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":873,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.547207+0000","last_change":"2024-09-17T07:48:53.510147+0000","last_active":"2024-09-17T07:49:14.547207+0000","last_peered":"2024-09-17T07:49:14.547207+0000","last_clean":"2024-09-17T07:48:50.494421+0000","last_became_active":"2024-09-17T07:48:53.510147+0000","last_became_peered":"2024-09-17T07:48:53.510147+0000","last_unstale":"2024-09-17T07:49:14.547207+0000","last_undegraded":"2024-09-17T07:48:53.509603+0000","last_fullsized":"2024-09-17T07:48:53.509541+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:52:15.656871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":756,"num_read_kb":27413,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-17T07:49:14.547168+0000","last_change":"2024-09-17T07:48:05.649758+0000","last_active":"2024-09-17T07:49:14.547168+0000","last_peered":"2024-09-17T07:49:14.547168+0000","last_clean":"2024-09-17T07:49:14.547168+0000","last_became_active":"2024-09-17T07:48:03.642170+0000","last_became_peered":"2024-09-17T07:48:03.642170+0000","last_unstale":"2024-09-17T07:49:14.547168+0000","last_undegraded":"2024-09-17T07:49:14.547168+0000","last_fullsized":"2024-09-17T07:49:14.547168+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:30.026257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"54'4","reported_seq":79,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979236+0000","last_change":"2024-09-17T07:48:02.160661+0000","last_active":"2024-09-17T07:48:49.979236+0000","last_peered":"2024-09-17T07:48:49.979236+0000","last_clean":"2024-09-17T07:47:42.119185+0000","last_became_active":"2024-09-17T07:47:45.604354+0000","last_became_peered":"2024-09-17T07:47:45.604354+0000","last_unstale":"2024-09-17T07:48:52.503859+0000","last_undegraded":"2024-09-17T07:48:02.140633+0000","last_fullsized":"2024-09-17T07:47:45.603702+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:54:04.231757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"58'257","reported_seq":1185,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.546410+0000","last_change":"2024-09-17T07:48:01.916903+0000","last_active":"2024-09-17T07:49:14.546410+0000","last_peered":"2024-09-17T07:49:14.546410+0000","last_clean":"2024-09-17T07:47:43.970152+0000","last_became_active":"2024-09-17T07:47:45.604092+0000","last_became_peered":"2024-09-17T07:47:45.604092+0000","last_unstale":"2024-09-17T07:49:14.546410+0000","last_undegraded":"2024-09-17T07:48:01.915803+0000","last_fullsized":"2024-09-17T07:47:45.603408+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:28.677664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797533,"num_objects":17,"num_object_clones":1,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1339,"num_read_kb":77312,"num_write":306,"num_write_kb":58504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":49,"reported_epoch":77,"state":"active+undersized","last_fresh":"2024-09-17T07:49:14.547377+0000","last_change":"2024-09-17T07:48:53.510166+0000","last_active":"2024-09-17T07:49:14.547377+0000","last_peered":"2024-09-17T07:49:14.547377+0000","last_clean":"2024-09-17T07:48:50.494734+0000","last_became_active":"2024-09-17T07:48:53.510166+0000","last_became_peered":"2024-09-17T07:48:53.510166+0000","last_unstale":"2024-09-17T07:49:14.547377+0000","last_undegraded":"2024-09-17T07:49:14.547377+0000","last_fullsized":"2024-09-17T07:48:53.509462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:33.475636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"45'5","reported_seq":55,"reported_epoch":77,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:14.546483+0000","last_change":"2024-09-17T07:48:04.736433+0000","last_active":"2024-09-17T07:49:14.546483+0000","last_peered":"2024-09-17T07:49:14.546483+0000","last_clean":"2024-09-17T07:49:14.546483+0000","last_became_active":"2024-09-17T07:48:04.643512+0000","last_became_peered":"2024-09-17T07:48:04.643512+0000","last_unstale":"2024-09-17T07:49:14.546483+0000","last_undegraded":"2024-09-17T07:49:14.546483+0000","last_fullsized":"2024-09-17T07:49:14.546483+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:58.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"71'184","reported_seq":926,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-17T07:49:16.510444+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:49:16.510444+0000","last_peered":"2024-09-17T07:49:16.510444+0000","last_clean":"2024-09-17T07:49:16.510444+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:49:16.510444+0000","last_undegraded":"2024-09-17T07:49:16.510444+0000","last_fullsized":"2024-09-17T07:49:16.510444+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:04:40.147948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777322,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1408,"num_read_kb":46186,"num_write":100,"num_write_kb":32777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":106,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+undersized","last_fresh":"2024-09-17T07:49:14.547480+0000","last_change":"2024-09-17T07:48:03.641341+0000","last_active":"2024-09-17T07:49:14.547480+0000","last_peered":"2024-09-17T07:49:14.547480+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.641341+0000","last_became_peered":"2024-09-17T07:48:03.641341+0000","last_unstale":"2024-09-17T07:49:14.547480+0000","last_undegraded":"2024-09-17T07:49:14.547480+0000","last_fullsized":"2024-09-17T07:48:03.640567+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:02.148375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":114,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.547146+0000","last_change":"2024-09-17T07:48:53.509681+0000","last_active":"2024-09-17T07:49:14.547146+0000","last_peered":"2024-09-17T07:49:14.547146+0000","last_clean":"2024-09-17T07:48:50.494470+0000","last_became_active":"2024-09-17T07:48:53.509681+0000","last_became_peered":"2024-09-17T07:48:53.509681+0000","last_unstale":"2024-09-17T07:49:14.547146+0000","last_undegraded":"2024-09-17T07:48:53.508978+0000","last_fullsized":"2024-09-17T07:48:53.508899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:56.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":132,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.547083+0000","last_change":"2024-09-17T07:48:09.653077+0000","last_active":"2024-09-17T07:49:14.547083+0000","last_peered":"2024-09-17T07:49:14.547083+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:48:09.653077+0000","last_became_peered":"2024-09-17T07:48:09.653077+0000","last_unstale":"2024-09-17T07:49:14.547083+0000","last_undegraded":"2024-09-17T07:48:09.652573+0000","last_fullsized":"2024-09-17T07:48:09.652495+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:35.420073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"55'78","reported_seq":215,"reported_epoch":77,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:14.546464+0000","last_change":"2024-09-17T07:48:04.691716+0000","last_active":"2024-09-17T07:49:14.546464+0000","last_peered":"2024-09-17T07:49:14.546464+0000","last_clean":"2024-09-17T07:49:14.546464+0000","last_became_active":"2024-09-17T07:48:04.643230+0000","last_became_peered":"2024-09-17T07:48:04.643230+0000","last_unstale":"2024-09-17T07:49:14.546464+0000","last_undegraded":"2024-09-17T07:49:14.546464+0000","last_fullsized":"2024-09-17T07:49:14.546464+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:16:12.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":88,"num_write_kb":24609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":27,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979315+0000","last_change":"2024-09-17T07:48:26.474017+0000","last_active":"2024-09-17T07:48:49.979315+0000","last_peered":"2024-09-17T07:48:49.979315+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640668+0000","last_became_peered":"2024-09-17T07:48:03.640668+0000","last_unstale":"2024-09-17T07:48:52.503860+0000","last_undegraded":"2024-09-17T07:48:25.341555+0000","last_fullsized":"2024-09-17T07:48:03.639432+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:44.698558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":106,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.547268+0000","last_change":"2024-09-17T07:48:53.509922+0000","last_active":"2024-09-17T07:49:14.547268+0000","last_peered":"2024-09-17T07:49:14.547268+0000","last_clean":"2024-09-17T07:48:50.494622+0000","last_became_active":"2024-09-17T07:48:53.509922+0000","last_became_peered":"2024-09-17T07:48:53.509922+0000","last_unstale":"2024-09-17T07:49:14.547268+0000","last_undegraded":"2024-09-17T07:48:53.509331+0000","last_fullsized":"2024-09-17T07:48:53.509273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:23.503950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":38,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979198+0000","last_change":"2024-09-17T07:48:25.422858+0000","last_active":"2024-09-17T07:48:49.979198+0000","last_peered":"2024-09-17T07:48:49.979198+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640271+0000","last_became_peered":"2024-09-17T07:48:03.640271+0000","last_unstale":"2024-09-17T07:48:52.503861+0000","last_undegraded":"2024-09-17T07:48:25.340395+0000","last_fullsized":"2024-09-17T07:48:03.639357+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:03.926215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":67,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.547189+0000","last_change":"2024-09-17T07:48:01.981238+0000","last_active":"2024-09-17T07:49:14.547189+0000","last_peered":"2024-09-17T07:49:14.547189+0000","last_clean":"2024-09-17T07:47:41.593808+0000","last_became_active":"2024-09-17T07:47:45.603610+0000","last_became_peered":"2024-09-17T07:47:45.603610+0000","last_unstale":"2024-09-17T07:49:14.547189+0000","last_undegraded":"2024-09-17T07:48:01.980889+0000","last_fullsized":"2024-09-17T07:47:45.603119+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:18:17.830983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":30,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979225+0000","last_change":"2024-09-17T07:48:04.643889+0000","last_active":"2024-09-17T07:48:49.979225+0000","last_peered":"2024-09-17T07:48:49.979225+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640398+0000","last_became_peered":"2024-09-17T07:48:03.640398+0000","last_unstale":"2024-09-17T07:48:52.503862+0000","last_undegraded":"2024-09-17T07:48:03.642853+0000","last_fullsized":"2024-09-17T07:48:03.639625+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:08.008675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":59,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.546418+0000","last_change":"2024-09-17T07:48:01.880716+0000","last_active":"2024-09-17T07:49:14.546418+0000","last_peered":"2024-09-17T07:49:14.546418+0000","last_clean":"2024-09-17T07:46:51.988421+0000","last_became_active":"2024-09-17T07:47:45.604099+0000","last_became_peered":"2024-09-17T07:47:45.604099+0000","last_unstale":"2024-09-17T07:49:14.546418+0000","last_undegraded":"2024-09-17T07:48:01.880450+0000","last_fullsized":"2024-09-17T07:47:45.603421+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:28.433406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":55,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.547067+0000","last_change":"2024-09-17T07:48:53.509737+0000","last_active":"2024-09-17T07:49:14.547067+0000","last_peered":"2024-09-17T07:49:14.547067+0000","last_clean":"2024-09-17T07:48:50.494367+0000","last_became_active":"2024-09-17T07:48:53.509737+0000","last_became_peered":"2024-09-17T07:48:53.509737+0000","last_unstale":"2024-09-17T07:49:14.547067+0000","last_undegraded":"2024-09-17T07:48:53.508926+0000","last_fullsized":"2024-09-17T07:48:53.508829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"45'5","reported_seq":66,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:14.546573+0000","last_change":"2024-09-17T07:48:53.508703+0000","last_active":"2024-09-17T07:49:14.546573+0000","last_peered":"2024-09-17T07:49:14.546573+0000","last_clean":"2024-09-17T07:48:02.144100+0000","last_became_active":"2024-09-17T07:48:53.508703+0000","last_became_peered":"2024-09-17T07:48:53.508703+0000","last_unstale":"2024-09-17T07:49:14.546573+0000","last_undegraded":"2024-09-17T07:48:53.508098+0000","last_fullsized":"2024-09-17T07:48:53.507992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:19.938198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":89,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":3},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80740871,"num_objects":43,"num_object_clones":1,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3717,"num_read_kb":175577,"num_write":530,"num_write_kb":132276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":316,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122744832,"data_stored":125829937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":564,"ondisk_log_size":564,"up":5,"acting":6,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":1,"acting":1,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217915,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":44,"num_write_kb":131084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213954560,"data_stored":213909821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":8,"acting":9,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640283,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388356,"kb_used_data":86836,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93983484,"statfs":{"total":96636764160,"available":96239087616,"internally_reserved":0,"allocated":88920064,"data_stored":88404740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57599999999999996}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54900000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.44},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.33700000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373108,"kb_used_data":71588,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93998732,"statfs":{"total":96636764160,"available":96254701568,"internally_reserved":0,"allocated":73306112,"data_stored":75876442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.625,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40699999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.254}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.316},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.32800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319040,"kb_used_data":17516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94052800,"statfs":{"total":96636764160,"available":96310067200,"internally_reserved":0,"allocated":17936384,"data_stored":17586425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.309}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.24299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640288,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355692,"kb_used_data":54172,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94016148,"statfs":{"total":96636764160,"available":96272535552,"internally_reserved":0,"allocated":55472128,"data_stored":54904861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.315,"15min":0.32700000000000001},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.314,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.215},{"interface":"front","average":{"1min":0.247,"5min":0.30299999999999999,"15min":0.312},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.35299999999999998,"5min":0.434,"15min":0.434},"last":0.23899999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.628,"15min":0.628},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.45500000000000002,"15min":0.46899999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.49399999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34599999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438}]}]},{"osd":1,"up_from":11,"seq":47244640289,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409580,"kb_used_data":103900,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93962260,"statfs":{"total":96636764160,"available":96217354240,"internally_reserved":0,"allocated":106393600,"data_stored":105826901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.441,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.35499999999999998,"15min":0.36099999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.498,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.27300000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61299999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.58099999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.374}]}]},{"osd":0,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301928,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069912,"statfs":{"total":96636764160,"available":96327589888,"internally_reserved":0,"allocated":421888,"data_stored":155934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16777216,"data_stored":16777216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75517952,"data_stored":75497640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29372416,"data_stored":29360350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29372416,"data_stored":29360310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51408896,"data_stored":54526249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:49:21.305 INFO:tasks.thrashosds.thrasher:Setting 4.2 to [3, 2, 4, 1] 2024-09-17T07:49:21.305 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.2', '3', '2', '4', '1'] 2024-09-17T07:49:21.305 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.2 3 2 4 1 2024-09-17T07:49:21.306 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:21.300+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:21.401 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:21.399+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:21.445 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:21.446 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:21.446 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:21.446 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:21.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:21.540 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:21.539+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:21.599 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:21.599 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:21.599 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:21.599 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:21.599 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:21.602 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:21.601+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:21.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:21.703 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:21.702+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:21.752 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:21.752 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:21.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:21.752 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:21.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.752 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.493236890999995, 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:21.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.124931274999994, 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:21.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:21.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.432749020000003, 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.762 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:21.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.426892163999995, 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.773 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.415199204999993, 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:21.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.346344622000004, 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:21.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.334445333000005, 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.785 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:21.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.790 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.283600785999994, 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:21.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:21.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.796 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.277592579, 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:21.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.271892222000005, 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:21.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.266152597000001, 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:21.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:21.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:21.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.933041548999995, 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:21.815 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:21.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 66.087940652, 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 66.081853929000005, 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.998397628000006, 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.992083195999996, 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.985711735999999, 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.979789260000004, 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:21.857 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 65.921806364000005, 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:21.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:21.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 61.964101692, 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:21.867 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:21.869 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:21.869 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:21.868+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:21.969 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:21.968+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:21.979 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:21.979 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:21.979 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:21.979 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:21.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:22.070 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:22.069+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:22.085 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:22.085 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:22.085 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:22.085 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:22.085 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:22.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:22.171 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:22.170+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.025272624999999, 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.017933291000006, 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.017216216999998, 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.007988655999995, 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.998764088000001, 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.983289447999994, 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.973999938000006, 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.967409692999993, 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.951227212000006, 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.942152292000003, 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.601713415999996, 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.599661139000006, 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.539429487999996, 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.378250339000004, 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.259 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:22.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.263 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.378226037999994, 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.264 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:22.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:22.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.378191662000006, 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:22.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:22.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.378170304999998, 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.275 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:22.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.375501661000001, 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:22.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:22.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:22.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.365784087999998, 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:22.285 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:22.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:22.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:22.290 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.399673698999997, 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:22.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:22.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:22.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:22.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.296 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:22.296 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:22.296 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:22.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:22.298 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:22.296+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:22.398 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:22.397+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:22.404 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:22.405 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:22.405 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:22.405 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:22.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:22.499 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:22.498+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:22.555 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:22.555 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:22.555 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:22.555 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:22.555 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:22.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:22.575 INFO:teuthology.orchestra.run.smithi026.stderr:set 4.2 pg_upmap_items mapping to [3->2,4->1] 2024-09-17T07:49:22.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:49:22.600 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:22.600+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.512225283999996, 2024-09-17T07:49:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.504565768000006, 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.496176199999994, 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.495804949999993, 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.487741985, 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.487370873000003, 2024-09-17T07:49:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.478090468000005, 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.476431035000004, 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.470406209999993, 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.470155883000004, 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.463042044999995, 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.462524946000002, 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.443553778999998, 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.437296919000005, 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.430760856999996, 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.057894446999995, 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:22.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:22.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.055934039999997, 2024-09-17T07:49:22.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:22.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:22.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:22.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.170265404000006, 2024-09-17T07:49:22.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:22.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.859419766000002, 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:22.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 69.858058321000001, 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:22.780 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:22.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:22.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:22.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:22.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:22.786 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:22.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:22.788 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:22.787+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:22.888 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:22.888+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:22.890 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:22.890 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:22.890 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:22.890 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:22.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:22.989 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:22.988+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:23.000 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:23.000 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:23.000 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:23.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:23.090 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:23.090+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:23.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:23.152 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:23.152 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.893763942999996, 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:23.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:23.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:23.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.525458326999996, 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:23.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:23.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.833276072000004, 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:23.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.827419215999996, 2024-09-17T07:49:23.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:23.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.815726256999994, 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:23.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:23.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:23.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.746871674000005, 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:23.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:23.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:23.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.734972385000006, 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.189 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:23.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:23.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.194 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.684127837999995, 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:23.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:23.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:23.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.678119631000001, 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:23.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:23.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.672419274000006, 2024-09-17T07:49:23.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:23.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:23.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:23.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.666679649000002, 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.212 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:23.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 70.333568600999996, 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:23.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:23.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:23.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.488467704000001, 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.225 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:23.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:23.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.482380981000006, 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:23.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:23.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.398924679999993, 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:23.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.392610247999997, 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:23.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.386238788, 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:23.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:23.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:23.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.380316312000005, 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:23.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:23.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:23.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.322333416000006, 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:23.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:23.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:23.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:23.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.364628744000001, 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:23.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:23.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:23.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:23.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:23.274 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:23.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:23.277 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:23.277 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:23.277 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:23.278 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:23.278 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:23.278 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:23.278 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:23.274+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:23.279 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:23.279 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:23.280 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:23.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:23.376 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:23.375+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:23.387 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:23.387 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:23.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:23.478 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:23.477+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:23.494 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:23.494 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:23.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:49:23.587 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:23.587+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:23.641 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:23.641 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:23.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:23.693 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:23.692+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:23.752 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:23.752 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:23.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:23.782 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:23.781+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:23.885 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:23.885+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:23.903 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:23.903 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:23.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:23.987 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:23.985+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:24.013 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.013 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:24.090 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:24.090+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:24.163 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.164 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:24.215 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:24.214+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:24.273 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.273 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:24.323 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:24.321+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:24.394 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:24.393+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:24.422 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.422 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:24.499 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:24.498+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:24.533 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.533 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:24.599 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:24.598+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:24.639 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.639 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:24.702 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:24.701+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.746 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:24.803 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:24.802+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:24.858 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:24.858 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:24.867 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:24.907 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:24.906+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.005 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:25.004+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.007 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:25.008 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:25.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:25.106 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:25.105+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.158 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:25.158 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:25.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:25.207 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:25.207+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:25.308 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:25.308+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:25.309 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:25.309 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:25.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:25.386 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:25.402 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:25.402 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:25.409 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:25.409+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:25.461 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:25.461 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:25.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:25.510 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:25.509+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.592 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:49:25.612 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:25.612 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:25.613 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:25.611+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:25.619 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:25.620 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:25.620 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:25.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:25.636 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:25.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:25.712 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:25.711+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.732 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:25.732 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:25.732 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:25.733 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:25.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:25.813 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:25.811+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:25.839 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:25.840 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:25.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:25.914 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:25.913+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:25.945 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:25.945 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.778890368000006, 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:25.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:25.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:25.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.771551033999998, 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:25.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:25.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.770833960000004, 2024-09-17T07:49:25.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:25.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:25.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.761606399000001, 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.961 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:25.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:25.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:25.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.752381830999994, 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:25.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.968 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:25.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.972 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.736907191, 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.973 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:25.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.727617680999998, 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:25.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:25.982 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.721027436, 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:25.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:25.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:25.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.704844954999999, 2024-09-17T07:49:25.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:25.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.695770034999995, 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:25.992 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:25.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:25.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.355331159000002, 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:25.997 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:25.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:25.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:26.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.003 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.353278881999998, 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:26.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:26.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:26.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.293047231000003, 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.010 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:26.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:26.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.015 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.131868081999997, 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:26.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:26.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.131843781000001, 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:26.021 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:26.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.026 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.131809404999998, 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:26.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:26.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.032 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.131788048000004, 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:26.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:26.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.129119403999994, 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.038 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:26.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:26.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.119401831000005, 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:26.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.153291441999997, 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:26.049 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:26.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:26.055 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:26.057 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:26.057 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:26.056+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:26.157 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:26.157+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:26.159 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:26.159 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:26.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:26.258 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:26.256+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:26.269 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:26.270 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:26.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:26.362 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:26.361+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:26.422 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:26.422 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:26.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:26.422 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:26.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.422 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.255401840000005, 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:26.428 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.247742324000001, 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.239352756000002, 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:26.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:26.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:26.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:26.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.238981506000002, 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.230918540999994, 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.230547428999998, 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.221267023999999, 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.219607590999999, 2024-09-17T07:49:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.213582766000002, 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.213332438999998, 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.475 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.206218601000003, 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:26.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.205701501999997, 2024-09-17T07:49:26.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:26.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:26.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:26.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.186730334999993, 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.180473474999999, 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.173937413000004, 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:26.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:26.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.801071003000004, 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.512 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:26.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:26.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.799110596000006, 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 77.91344196, 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.602596321999997, 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:26.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:26.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 73.601234876999996, 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.547 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:26.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:26.549 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:26.548+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:49:26.649 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:26.649+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:26.687 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:26.687 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:26.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:26.749 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:26.748+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:26.798 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:26.799 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:26.799 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:26.799 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:26.799 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:26.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:26.851 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:26.850+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.692524129999995, 2024-09-17T07:49:26.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:26.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:26.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.324218513999995, 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:26.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.632036259000003, 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:26.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:26.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:26.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.626179402999995, 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.970 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:26.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:26.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:26.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.614486443999994, 2024-09-17T07:49:26.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:26.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:26.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:26.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.545631861000004, 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:26.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:26.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.533732572000005, 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.482888024999994, 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.005 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.476879818, 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:27.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:27.012 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.471179461000006, 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:27.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.019 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.465439836000002, 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:27.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:27.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:27.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:27.026 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.132328787999995, 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:27.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:27.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:27.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.287227891000001, 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:27.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:27.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.281141168000005, 2024-09-17T07:49:27.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:27.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:27.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:27.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.197684867000007, 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:27.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:27.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:27.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:27.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:27.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.191370434999996, 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:27.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:27.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:27.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:27.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.184998974999999, 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:27.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:27.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:27.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.179076499000004, 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:27.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:27.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:27.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.121093603000006, 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:27.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:27.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:27.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.163388931, 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:27.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:27.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:27.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.086 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:27.087 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:26.950+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:27.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:27.089 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:27.089+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:27.190 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:27.190+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:27.198 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:27.199 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:27.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:27.292 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:27.291+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:27.347 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:27.347 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:27.347 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:27.347 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:27.347 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:27.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:27.393 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:27.392+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.287532236000004, 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:27.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:27.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.280192901999996, 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:27.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.279475828000002, 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:27.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:27.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:27.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.270248266999999, 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:27.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:27.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.261023699000006, 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.475 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:27.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:27.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:27.479 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.245549058999998, 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.480 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.236259548999996, 2024-09-17T07:49:27.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:27.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:27.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:27.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:27.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.229669303999998, 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:27.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:27.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.213486822999997, 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:27.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:27.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:27.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:27.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:27.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:27.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:27.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:27.503 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.204411902999993, 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:27.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:27.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:27.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.863973027, 2024-09-17T07:49:27.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:27.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:27.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:27.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.861920749999996, 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:27.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:27.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:27.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:27.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:27.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:27.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:27.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:27.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:27.524 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.801689099000001, 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:27.525 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:27.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.526 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:27.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:27.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:27.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.640509949999995, 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.533 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:27.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.640485648999999, 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.640451272999996, 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:27.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.640429916000002, 2024-09-17T07:49:27.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:27.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:27.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:27.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.557 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.558 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.558 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.559 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:27.559 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.637761272000006, 2024-09-17T07:49:27.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:27.559 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.559 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:27.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.560 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:27.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.560 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:27.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:27.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:27.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:27.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:27.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:27.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:27.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:27.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:27.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:27.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:27.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:27.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:27.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:27.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:27.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:27.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.570 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.628043699000003, 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:27.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:27.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.661933309999995, 2024-09-17T07:49:27.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:27.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:27.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:27.582 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:27.585 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:27.583+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:27.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:27.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:27.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:27.589 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T07:49:27.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:49:27.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:27.684 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:27.684+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:27.700 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:27.701 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:27.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:27.780 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:27.785 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:27.785+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:27.795 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:27.796 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:27.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:27.814 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:27.814 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:27.829 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:27.830 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:27.830 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:27.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:27.856 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:27.857 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:27.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:27.906 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:27.905+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:27.913 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 83, 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:49:26.581422+0000", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:48:52.499074+0000", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:46:43.251109+0000", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:49:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 4, 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:49:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:49:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:49:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:49:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "new_default_pool27356", 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:48:02.539579+0000", 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "49", 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 49, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 29, 2024-09-17T07:49:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/31884", 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/31884", 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/31884", 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/31884", 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 69, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 69, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:27.949 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:27.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:49:27.953 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.3944549560546875, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:27.954 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.957 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/31604", 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/31604", 2024-09-17T07:49:27.958 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/31604", 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/31604", 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.456390380859375, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:27.959 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:49:27.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 46, 2024-09-17T07:49:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/31606", 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/31606", 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/31606", 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/31606", 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:27.974 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.2", 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [ 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "snaps": [ 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "begin": 2, 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "length": 1 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:49:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:27.979 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:27.981 INFO:tasks.thrashosds.thrasher:j is {'epoch': 83, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:49:26.581422+0000', 'last_up_change': '2024-09-17T07:48:52.499074+0000', 'last_in_change': '2024-09-17T07:46:43.251109+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 29, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 31884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 31884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 31884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 31884}]}, 'public_addr': '172.21.15.26:6801/31884', 'cluster_addr': '172.21.15.26:6803/31884', 'heartbeat_back_addr': '172.21.15.26:6807/31884', 'heartbeat_front_addr': '172.21.15.26:6805/31884', 'state': ['exists']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 0, 'in': 1, 'weight': 0.3944549560546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 31604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 31604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 31604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 31604}]}, 'public_addr': '172.21.15.94:6801/31604', 'cluster_addr': '172.21.15.94:6803/31604', 'heartbeat_back_addr': '172.21.15.94:6807/31604', 'heartbeat_front_addr': '172.21.15.94:6805/31604', 'state': ['exists']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 0.456390380859375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 31606}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 31606}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 31606}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 31606}]}, 'public_addr': '172.21.15.94:6817/31606', 'cluster_addr': '172.21.15.94:6819/31606', 'heartbeat_back_addr': '172.21.15.94:6823/31606', 'heartbeat_front_addr': '172.21.15.94:6821/31606', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '1.2', 'osds': [2, 4]}, {'pgid': '3.1', 'osds': [2, 4]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:49:27.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:49:27.986 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:27.986+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:28.007 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.007 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:28.087 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:28.087+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:28.160 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.160 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:28.188 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:28.188+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:28.271 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.271 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:28.283 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:49:28.295 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-17T07:49:27.439951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215549224,"num_objects":96,"num_object_clones":1,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":6,"num_read":3894,"num_read_kb":175713,"num_write":681,"num_write_kb":264994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165881,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":316,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":692,"ondisk_log_size":692,"up":23,"acting":25,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2147840,"kb_used_data":334568,"kb_used_omap":9,"kb_used_meta":1813174,"kb_avail":564083200,"statfs":{"total":579820584960,"available":577621196800,"internally_reserved":0,"allocated":342597632,"data_stored":342820550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1856690192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001444"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":26,"reported_epoch":67,"state":"stale+active+undersized","last_fresh":"2024-09-17T07:48:49.979339+0000","last_change":"2024-09-17T07:48:03.640213+0000","last_active":"2024-09-17T07:48:49.979339+0000","last_peered":"2024-09-17T07:48:49.979339+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640213+0000","last_became_peered":"2024-09-17T07:48:03.640213+0000","last_unstale":"2024-09-17T07:48:52.503858+0000","last_undegraded":"2024-09-17T07:48:49.979339+0000","last_fullsized":"2024-09-17T07:48:03.639399+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:21.591427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"45'7","reported_seq":80,"reported_epoch":48,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:05.372036+0000","last_change":"2024-09-17T07:48:01.879165+0000","last_active":"2024-09-17T07:48:05.372036+0000","last_peered":"2024-09-17T07:48:05.372036+0000","last_clean":"2024-09-17T07:47:41.593387+0000","last_became_active":"2024-09-17T07:47:45.602779+0000","last_became_peered":"2024-09-17T07:47:45.602779+0000","last_unstale":"2024-09-17T07:48:08.649038+0000","last_undegraded":"2024-09-17T07:48:01.878970+0000","last_fullsized":"2024-09-17T07:47:45.602289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:18.255015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":878,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578551+0000","last_change":"2024-09-17T07:48:53.510147+0000","last_active":"2024-09-17T07:49:23.578551+0000","last_peered":"2024-09-17T07:49:23.578551+0000","last_clean":"2024-09-17T07:48:50.494421+0000","last_became_active":"2024-09-17T07:48:53.510147+0000","last_became_peered":"2024-09-17T07:48:53.510147+0000","last_unstale":"2024-09-17T07:49:23.578551+0000","last_undegraded":"2024-09-17T07:48:53.509603+0000","last_fullsized":"2024-09-17T07:48:53.509541+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:52:15.656871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":756,"num_read_kb":27413,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":50,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-17T07:49:23.578317+0000","last_change":"2024-09-17T07:48:05.649758+0000","last_active":"2024-09-17T07:49:23.578317+0000","last_peered":"2024-09-17T07:49:23.578317+0000","last_clean":"2024-09-17T07:49:23.578317+0000","last_became_active":"2024-09-17T07:48:03.642170+0000","last_became_peered":"2024-09-17T07:48:03.642170+0000","last_unstale":"2024-09-17T07:49:23.578317+0000","last_undegraded":"2024-09-17T07:49:23.578317+0000","last_fullsized":"2024-09-17T07:49:23.578317+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:30.026257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"54'4","reported_seq":79,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979236+0000","last_change":"2024-09-17T07:48:02.160661+0000","last_active":"2024-09-17T07:48:49.979236+0000","last_peered":"2024-09-17T07:48:49.979236+0000","last_clean":"2024-09-17T07:47:42.119185+0000","last_became_active":"2024-09-17T07:47:45.604354+0000","last_became_peered":"2024-09-17T07:47:45.604354+0000","last_unstale":"2024-09-17T07:48:52.503859+0000","last_undegraded":"2024-09-17T07:48:02.140633+0000","last_fullsized":"2024-09-17T07:47:45.603702+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:54:04.231757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"58'257","reported_seq":1190,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578497+0000","last_change":"2024-09-17T07:48:01.916903+0000","last_active":"2024-09-17T07:49:23.578497+0000","last_peered":"2024-09-17T07:49:23.578497+0000","last_clean":"2024-09-17T07:47:43.970152+0000","last_became_active":"2024-09-17T07:47:45.604092+0000","last_became_peered":"2024-09-17T07:47:45.604092+0000","last_unstale":"2024-09-17T07:49:23.578497+0000","last_undegraded":"2024-09-17T07:48:01.915803+0000","last_fullsized":"2024-09-17T07:47:45.603408+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:28.677664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797533,"num_objects":17,"num_object_clones":1,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1339,"num_read_kb":77312,"num_write":306,"num_write_kb":58504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":54,"reported_epoch":82,"state":"active+undersized","last_fresh":"2024-09-17T07:49:23.578355+0000","last_change":"2024-09-17T07:48:53.510166+0000","last_active":"2024-09-17T07:49:23.578355+0000","last_peered":"2024-09-17T07:49:23.578355+0000","last_clean":"2024-09-17T07:48:50.494734+0000","last_became_active":"2024-09-17T07:48:53.510166+0000","last_became_peered":"2024-09-17T07:48:53.510166+0000","last_unstale":"2024-09-17T07:49:23.578355+0000","last_undegraded":"2024-09-17T07:49:23.578355+0000","last_fullsized":"2024-09-17T07:48:53.509462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:33.475636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"45'5","reported_seq":60,"reported_epoch":82,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:23.578107+0000","last_change":"2024-09-17T07:48:04.736433+0000","last_active":"2024-09-17T07:49:23.578107+0000","last_peered":"2024-09-17T07:49:23.578107+0000","last_clean":"2024-09-17T07:49:23.578107+0000","last_became_active":"2024-09-17T07:48:04.643512+0000","last_became_peered":"2024-09-17T07:48:04.643512+0000","last_unstale":"2024-09-17T07:49:23.578107+0000","last_undegraded":"2024-09-17T07:49:23.578107+0000","last_fullsized":"2024-09-17T07:49:23.578107+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:58.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"83'185","reported_seq":941,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-17T07:49:26.722607+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:49:26.722607+0000","last_peered":"2024-09-17T07:49:26.722607+0000","last_clean":"2024-09-17T07:49:26.722607+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:49:26.722607+0000","last_undegraded":"2024-09-17T07:49:26.722607+0000","last_fullsized":"2024-09-17T07:49:26.722607+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:04:40.147948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777322,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1410,"num_read_kb":46187,"num_write":100,"num_write_kb":32777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":106,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+undersized","last_fresh":"2024-09-17T07:49:23.578650+0000","last_change":"2024-09-17T07:48:03.641341+0000","last_active":"2024-09-17T07:49:23.578650+0000","last_peered":"2024-09-17T07:49:23.578650+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.641341+0000","last_became_peered":"2024-09-17T07:48:03.641341+0000","last_unstale":"2024-09-17T07:49:23.578650+0000","last_undegraded":"2024-09-17T07:49:23.578650+0000","last_fullsized":"2024-09-17T07:48:03.640567+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:02.148375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":119,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578495+0000","last_change":"2024-09-17T07:48:53.509681+0000","last_active":"2024-09-17T07:49:23.578495+0000","last_peered":"2024-09-17T07:49:23.578495+0000","last_clean":"2024-09-17T07:48:50.494470+0000","last_became_active":"2024-09-17T07:48:53.509681+0000","last_became_peered":"2024-09-17T07:48:53.509681+0000","last_unstale":"2024-09-17T07:49:23.578495+0000","last_undegraded":"2024-09-17T07:48:53.508978+0000","last_fullsized":"2024-09-17T07:48:53.508899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:56.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":137,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578407+0000","last_change":"2024-09-17T07:48:09.653077+0000","last_active":"2024-09-17T07:49:23.578407+0000","last_peered":"2024-09-17T07:49:23.578407+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:48:09.653077+0000","last_became_peered":"2024-09-17T07:48:09.653077+0000","last_unstale":"2024-09-17T07:49:23.578407+0000","last_undegraded":"2024-09-17T07:48:09.652573+0000","last_fullsized":"2024-09-17T07:48:09.652495+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:35.420073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"55'78","reported_seq":220,"reported_epoch":82,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:23.578107+0000","last_change":"2024-09-17T07:48:04.691716+0000","last_active":"2024-09-17T07:49:23.578107+0000","last_peered":"2024-09-17T07:49:23.578107+0000","last_clean":"2024-09-17T07:49:23.578107+0000","last_became_active":"2024-09-17T07:48:04.643230+0000","last_became_peered":"2024-09-17T07:48:04.643230+0000","last_unstale":"2024-09-17T07:49:23.578107+0000","last_undegraded":"2024-09-17T07:49:23.578107+0000","last_fullsized":"2024-09-17T07:49:23.578107+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:16:12.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":88,"num_write_kb":24609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":27,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979315+0000","last_change":"2024-09-17T07:48:26.474017+0000","last_active":"2024-09-17T07:48:49.979315+0000","last_peered":"2024-09-17T07:48:49.979315+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640668+0000","last_became_peered":"2024-09-17T07:48:03.640668+0000","last_unstale":"2024-09-17T07:48:52.503860+0000","last_undegraded":"2024-09-17T07:48:25.341555+0000","last_fullsized":"2024-09-17T07:48:03.639432+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:44.698558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":111,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578453+0000","last_change":"2024-09-17T07:48:53.509922+0000","last_active":"2024-09-17T07:49:23.578453+0000","last_peered":"2024-09-17T07:49:23.578453+0000","last_clean":"2024-09-17T07:48:50.494622+0000","last_became_active":"2024-09-17T07:48:53.509922+0000","last_became_peered":"2024-09-17T07:48:53.509922+0000","last_unstale":"2024-09-17T07:49:23.578453+0000","last_undegraded":"2024-09-17T07:48:53.509331+0000","last_fullsized":"2024-09-17T07:48:53.509273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:23.503950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":38,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979198+0000","last_change":"2024-09-17T07:48:25.422858+0000","last_active":"2024-09-17T07:48:49.979198+0000","last_peered":"2024-09-17T07:48:49.979198+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640271+0000","last_became_peered":"2024-09-17T07:48:03.640271+0000","last_unstale":"2024-09-17T07:48:52.503861+0000","last_undegraded":"2024-09-17T07:48:25.340395+0000","last_fullsized":"2024-09-17T07:48:03.639357+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:03.926215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":72,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578520+0000","last_change":"2024-09-17T07:48:01.981238+0000","last_active":"2024-09-17T07:49:23.578520+0000","last_peered":"2024-09-17T07:49:23.578520+0000","last_clean":"2024-09-17T07:47:41.593808+0000","last_became_active":"2024-09-17T07:47:45.603610+0000","last_became_peered":"2024-09-17T07:47:45.603610+0000","last_unstale":"2024-09-17T07:49:23.578520+0000","last_undegraded":"2024-09-17T07:48:01.980889+0000","last_fullsized":"2024-09-17T07:47:45.603119+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:18:17.830983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":30,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979225+0000","last_change":"2024-09-17T07:48:04.643889+0000","last_active":"2024-09-17T07:48:49.979225+0000","last_peered":"2024-09-17T07:48:49.979225+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640398+0000","last_became_peered":"2024-09-17T07:48:03.640398+0000","last_unstale":"2024-09-17T07:48:52.503862+0000","last_undegraded":"2024-09-17T07:48:03.642853+0000","last_fullsized":"2024-09-17T07:48:03.639625+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:08.008675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":64,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578511+0000","last_change":"2024-09-17T07:48:01.880716+0000","last_active":"2024-09-17T07:49:23.578511+0000","last_peered":"2024-09-17T07:49:23.578511+0000","last_clean":"2024-09-17T07:46:51.988421+0000","last_became_active":"2024-09-17T07:47:45.604099+0000","last_became_peered":"2024-09-17T07:47:45.604099+0000","last_unstale":"2024-09-17T07:49:23.578511+0000","last_undegraded":"2024-09-17T07:48:01.880450+0000","last_fullsized":"2024-09-17T07:47:45.603421+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:28.433406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":60,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578317+0000","last_change":"2024-09-17T07:48:53.509737+0000","last_active":"2024-09-17T07:49:23.578317+0000","last_peered":"2024-09-17T07:49:23.578317+0000","last_clean":"2024-09-17T07:48:50.494367+0000","last_became_active":"2024-09-17T07:48:53.509737+0000","last_became_peered":"2024-09-17T07:48:53.509737+0000","last_unstale":"2024-09-17T07:49:23.578317+0000","last_undegraded":"2024-09-17T07:48:53.508926+0000","last_fullsized":"2024-09-17T07:48:53.508829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"45'5","reported_seq":71,"reported_epoch":82,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:23.578218+0000","last_change":"2024-09-17T07:48:53.508703+0000","last_active":"2024-09-17T07:49:23.578218+0000","last_peered":"2024-09-17T07:49:23.578218+0000","last_clean":"2024-09-17T07:48:02.144100+0000","last_became_active":"2024-09-17T07:48:53.508703+0000","last_became_peered":"2024-09-17T07:48:53.508703+0000","last_unstale":"2024-09-17T07:49:23.578218+0000","last_undegraded":"2024-09-17T07:48:53.508098+0000","last_fullsized":"2024-09-17T07:48:53.507992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:19.938198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":89,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":3},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80740871,"num_objects":43,"num_object_clones":1,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3719,"num_read_kb":175578,"num_write":530,"num_write_kb":132276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":316,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122744832,"data_stored":125829937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":565,"ondisk_log_size":565,"up":5,"acting":6,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":1,"acting":1,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217915,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":44,"num_write_kb":131084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213954560,"data_stored":213909821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":8,"acting":9,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640283,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388356,"kb_used_data":86836,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93983484,"statfs":{"total":96636764160,"available":96239087616,"internally_reserved":0,"allocated":88920064,"data_stored":88404740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57599999999999996}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54900000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.44},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.33700000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373152,"kb_used_data":71636,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93998688,"statfs":{"total":96636764160,"available":96254656512,"internally_reserved":0,"allocated":73355264,"data_stored":75898191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.625,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42699999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34499999999999997}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.254}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.316},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.32800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319040,"kb_used_data":17516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94052800,"statfs":{"total":96636764160,"available":96310067200,"internally_reserved":0,"allocated":17936384,"data_stored":17586425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.309}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.24299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640290,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355736,"kb_used_data":54220,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94016104,"statfs":{"total":96636764160,"available":96272490496,"internally_reserved":0,"allocated":55521280,"data_stored":54926610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.315,"15min":0.32700000000000001},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.314,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.218},{"interface":"front","average":{"1min":0.247,"5min":0.30299999999999999,"15min":0.312},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.35299999999999998,"5min":0.434,"15min":0.434},"last":0.253}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.628,"15min":0.628},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.45500000000000002,"15min":0.46899999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.49399999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.35099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438}]}]},{"osd":1,"up_from":11,"seq":47244640291,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409628,"kb_used_data":103948,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93962212,"statfs":{"total":96636764160,"available":96217305088,"internally_reserved":0,"allocated":106442752,"data_stored":105848650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.441,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.35499999999999998,"15min":0.36099999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.498,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.25900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61299999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.58099999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38300000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.374}]}]},{"osd":0,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301928,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069912,"statfs":{"total":96636764160,"available":96327589888,"internally_reserved":0,"allocated":421888,"data_stored":155934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16777216,"data_stored":16777216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75517952,"data_stored":75497640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29372416,"data_stored":29360350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29372416,"data_stored":29360310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51408896,"data_stored":54526249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:49:28.297 INFO:tasks.thrashosds.thrasher:Setting 3.1 to [3, 2] 2024-09-17T07:49:28.297 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1', '3', '2'] 2024-09-17T07:49:28.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1 3 2 2024-09-17T07:49:28.297 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:28.293+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:28.382 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.382 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.389 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:28.389+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:28.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:28.490 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:28.490+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:28.496 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.496 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:28.591 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:28.591+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:28.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:49:28.602 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.602 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:28.692 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:28.691+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:28.709 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.709 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.718 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:28.792 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:28.792+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:28.819 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.820 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:28.912 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:28.912+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:28.969 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:28.970 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:28.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:29.019 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:29.019+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:29.095 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:29.095+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:29.120 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.120 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.128 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:29.212 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:29.212+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:29.271 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.272 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:29.298 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:29.298+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:29.398 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:29.398+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:29.422 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.422 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:29.513 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:29.513+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:29.571 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.572 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:29.596 INFO:teuthology.orchestra.run.smithi026.stderr:set 3.1 pg_upmap mapping to [3,2] 2024-09-17T07:49:29.596 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:49:29.600 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:29.600+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:29.689 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.690 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.698 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:29.701 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:29.701+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:29.797 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.797 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.802 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:29.802+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:29.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:29.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:29.903 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:29.903+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:29.904 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:29.904 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:29.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:29.952 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:29.968 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:29.984 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:29.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:30.017 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:30.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:30.017 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:30.017 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:30.017+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:30.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:30.109 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:30.109+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:30.167 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:30.167 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:30.167 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:30.167 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:30.168 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:30.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:30.220 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:30.217+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:30.306 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:30.305+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:30.320 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.153939207999997, 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.146279691999993, 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.137890123999995, 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.137518873999994, 2024-09-17T07:49:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.129455909000001, 2024-09-17T07:49:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.129084797000004, 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.119804392000006, 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.118144959000006, 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.112120133999994, 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.111869807000005, 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.104755968999996, 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:30.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:30.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.104238870000003, 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.085267703, 2024-09-17T07:49:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.079010843000006, 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.072474780999997, 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:30.415 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.699608370999997, 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:30.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:30.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:30.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:30.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.423 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:30.424 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.697647963999998, 2024-09-17T07:49:30.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:30.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.424 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:30.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:30.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.811979328000007, 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:30.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:30.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:30.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:30.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:30.436 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 77.501133690000003, 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:30.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 77.499772245000003, 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.443 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:30.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:30.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.448 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.449 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:30.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:30.451 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:30.449+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:30.551 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:30.551+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:30.593 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:30.594 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:30.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:30.687 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:30.687+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:30.744 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:30.744 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:30.744 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:30.744 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:30.744 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:30.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:30.753 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:30.752+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:30.853 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:30.853+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:30.896 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.637463385999993, 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 83.269157770000007, 2024-09-17T07:49:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.576975515000001, 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.571118659000007, 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:30.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.559425700000006, 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:30.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:30.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:30.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:30.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:30.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.490571117000002, 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:30.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:30.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:30.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.478671828000003, 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.427827281000006, 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.421819073999998, 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.946 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:30.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:30.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:30.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.416118717000003, 2024-09-17T07:49:30.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:30.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:30.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:30.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.410379091999999, 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:30.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:30.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:30.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.077268043999993, 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:30.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:30.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:30.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:30.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:30.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:30.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:30.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:30.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:30.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:30.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:30.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:30.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:30.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:30.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:30.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:30.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:30.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:30.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:30.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.978 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:30.978 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:30.978 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.232167146999998, 2024-09-17T07:49:30.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:30.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.980 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:30.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:30.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:30.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:30.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:30.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:30.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:30.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:30.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:30.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:30.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:30.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:30.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:30.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:30.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:30.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:30.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:30.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:30.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:30.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:30.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:30.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:30.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:30.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:30.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:30.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:30.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:30.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:30.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.226080424000003, 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:30.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:30.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:30.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:30.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:30.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:30.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:30.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:30.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:30.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:30.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:30.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:30.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:31.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:31.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:31.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:31.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:31.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:31.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.142624123000004, 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:31.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:31.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.136309690999994, 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:31.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:31.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:31.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.129938230999997, 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.124015755000002, 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 75.066032859000003, 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 71.108328186999998, 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:31.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.040 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:31.041 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:31.042 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:31.040+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:31.142 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:31.142+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:31.150 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:31.151 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:31.151 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:31.151 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:31.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:31.243 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:31.243+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:31.299 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:31.299 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:31.299 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:31.308 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:31.344 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:31.344+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.240501828999996, 2024-09-17T07:49:31.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:31.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.233162495000002, 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.232445420999994, 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:31.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:31.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.421 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.223217860000005, 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:31.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:31.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:31.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.213993291999998, 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:31.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:31.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:31.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.198518652000004, 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.432 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:31.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.436 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.189229142000002, 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:31.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:31.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.182638897000004, 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:31.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.166456416000003, 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:31.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:31.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:31.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:31.450 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.157381495999999, 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:31.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:31.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.816942620000006, 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:31.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.814890343000002, 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:31.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 83.754658692000007, 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:31.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.593479543000001, 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:31.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:31.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.477 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.593455242000005, 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.478 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:31.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:31.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.593420866000002, 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:31.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:31.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.593399508999994, 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:31.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:31.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.590730864999998, 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:31.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:31.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:31.498 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.581013291999994, 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:31.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.614902903000001, 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:31.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:31.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:31.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:31.510 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:31.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:31.512 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:31.510+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:31.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:49:31.612 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:31.612+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:31.622 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:31.622 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:31.622 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:31.622 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:31.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:31.713 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:31.713+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:31.775 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:31.775 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:31.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:31.814 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:31.814+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.887 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.720487242999994, 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.712827727000004, 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:31.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:31.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:31.897 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.704438159000006, 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:31.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.704066909000005, 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:31.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:31.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.696003943999997, 2024-09-17T07:49:31.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:31.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:31.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.695632832000001, 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.686352427000003, 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:31.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:31.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.684692994000002, 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.678668169000005, 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.678417842000002, 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:31.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.671304004000007, 2024-09-17T07:49:31.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:31.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.942 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.942 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:31.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:31.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:31.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:31.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:31.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:31.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:31.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:31.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:31.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:31.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:31.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.953 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.953 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:31.953 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.670786905, 2024-09-17T07:49:31.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:31.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:31.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:31.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:31.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:31.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:31.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:31.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:31.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:31.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:31.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:31.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:31.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:31.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:31.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:31.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:31.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:31.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:31.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:31.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:31.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:31.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:31.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:31.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:31.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.651815737999996, 2024-09-17T07:49:31.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:31.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:31.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:31.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:31.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:31.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:31.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:31.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:31.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:31.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:31.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:31.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.975 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.975 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:31.976 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.645558878000003, 2024-09-17T07:49:31.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:31.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.976 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:31.977 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:31.977 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:31.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.977 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:31.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:31.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:31.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:31.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:31.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.639022815999994, 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:31.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.266156405999993, 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:31.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:31.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:31.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.264195998999995, 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 83.378527363000003, 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.067681725, 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.066320279999999, 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:32.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:32.020 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:32.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:32.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:32.022 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:32.020+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:32.121 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:32.121+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:32.164 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:32.167 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:32.167 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:32.179 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:32.186 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:32.186 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:32.202 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:32.202 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:32.202 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:32.218 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:32.218 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:32.218 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:32.222 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:32.222 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:32.221+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:32.287 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:32.288 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:32.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:32.339 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:32.339+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:32.397 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:32.397 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:32.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:32.449 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:32.449+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:32.515 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:32.515 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:32.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:32.525 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:32.524+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:32.598 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:49:32.625 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:32.625+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:32.665 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:32.666 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:32.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:32.727 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:32.727+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:32.813 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:32.813 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:32.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:32.865 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:32.865+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:32.924 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:32.924 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:32.929 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:32.928+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:32.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:33.029 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:33.029+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:33.075 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.075 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:33.131 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:33.131+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:33.186 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.186 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:33.238 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:33.238+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:33.296 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.296 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.304 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:33.332 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:33.331+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:33.433 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:33.433+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:33.446 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.447 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:33.534 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:33.533+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.556 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:33.635 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:33.634+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:33.666 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.667 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:33.735 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:33.735+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:33.774 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.774 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:33.867 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:33.867+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:33.920 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:33.921 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:33.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:33.973 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:33.973+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:34.034 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:34.034 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:34.038 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:34.038+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:34.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:34.138 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:34.138+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:34.185 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:34.185 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:34.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:34.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:34.239 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:34.239+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:34.338 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:34.338 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:34.340 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:34.340+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:34.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:34.352 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:34.368 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:34.384 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:34.384 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:34.384 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:34.388 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:34.440 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:34.440+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:34.453 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:34.453 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:34.453 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:34.454 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:34.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:34.546 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:34.546+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:34.604 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:34.604 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:34.605 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:34.605 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:34.605 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:34.611 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:34.611 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:34.611 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-17T07:49:34.611 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:49:34.611 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:49:34.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-09-17T07:49:34.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:49:34.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:34.656 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:34.655+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:34.721 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:34.721 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "" 2024-09-17T07:49:34.721 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:34.743 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:34.742+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:34.759 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.500878174999997, 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:34.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:34.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.132572558999996, 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:34.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:34.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.440390304000005, 2024-09-17T07:49:34.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:34.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.776 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.434533447999996, 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.422840488999995, 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:34.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:34.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:34.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.353985906000005, 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:34.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:34.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.342086617000007, 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:34.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.291242069999996, 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.801 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:34.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:34.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.285233863000002, 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.279533506000007, 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.273793881000003, 2024-09-17T07:49:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:34.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.940682832999997, 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:34.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:34.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.095581936000002, 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.089495213000006, 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 79.006038911999994, 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.999724479999998, 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:34.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.993353020000001, 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:34.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:34.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.987430544000006, 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:34.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:34.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 78.929447648000007, 2024-09-17T07:49:34.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:34.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:34.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:34.878 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 74.971742976000002, 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:34.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:34.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:34.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:34.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:34.883 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:34.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:34.885 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:34.883+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:34.984 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:34.984 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:34.984 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:34.984 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:34.985 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:34.984+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:34.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:35.085 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:35.085+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:35.129 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:35.129 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:35.129 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:35.130 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:35.130 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:35.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:35.186 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:35.185+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:35.278 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:35.278 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:35.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:35.278 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:35.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.278 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.111558454000004, 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:35.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:35.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.104219119999996, 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:35.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:35.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.103502046000003, 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:35.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:35.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:35.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.094274485, 2024-09-17T07:49:35.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.296 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.085049917000006, 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:35.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:35.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.069575276999998, 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:35.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.060285766999996, 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.305 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:35.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.053695521999998, 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:35.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:35.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:35.314 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.037513040999997, 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.318 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.028438120999994, 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:35.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.687999245, 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:35.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:35.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:35.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:35.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.685946967999996, 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:35.329 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:35.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.334 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.625715317000001, 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:35.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:35.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:35.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.464536167999995, 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.341 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:35.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:35.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.464511866999999, 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:35.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.464477490999997, 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.351 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:35.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.464456134000002, 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:35.356 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:35.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:35.361 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.461787490000006, 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:35.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:35.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:35.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.452069917000003, 2024-09-17T07:49:35.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:35.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.372 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.485959527999995, 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:35.373 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:35.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:35.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:35.379 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:35.381 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:35.380+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:35.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:35.481 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:35.481+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:35.488 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:35.489 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:35.489 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:35.489 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:35.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:35.583 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:35.583+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:35.600 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:35.600 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:35.600 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:35.600 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:35.600 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:35.603 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:49:35.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:35.684 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:35.683+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:35.713 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.546875979000006, 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:35.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.539216463000002, 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:35.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:35.726 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.530826895000004, 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.530455645000004, 2024-09-17T07:49:35.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:35.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.522392679999996, 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.522021568, 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.512741163000001, 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.511081730000001, 2024-09-17T07:49:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.505056905000004, 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.504806578, 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.497692740000005, 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.497175640999998, 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.478204473999995, 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.471947614000001, 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.465411552000006, 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.092545142000006, 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.090584734999993, 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.204916099000002, 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.894070460999998, 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 82.892709015999998, 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:35.826 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:35.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:35.828 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:35.827+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:35.928 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:35.928+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:35.970 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:35.970 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:35.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:36.062 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:36.062+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:36.119 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:36.119 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:36.120 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:36.120 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:36.120 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:36.128 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:36.130 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:36.130+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:36.231 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:36.231+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:36.272 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 94.013664766000005, 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.645359150000004, 2024-09-17T07:49:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.953176894999999, 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.947320039000004, 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.935627080000003, 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.866772496999999, 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.854873208000001, 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:36.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:36.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.804028661000004, 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.312 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:36.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:36.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.798020453999996, 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.318 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:36.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:36.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.792320097000001, 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.324 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:36.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.786580471999997, 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:36.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:36.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:36.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 83.453469424000005, 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:36.336 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:36.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:36.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.608368526999996, 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:36.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:36.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:36.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.602281804, 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.348 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:36.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:36.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.518825503000002, 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.359 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.512511071000006, 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.506139610999995, 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.365 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:36.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.500217135, 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.371 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:36.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.442234239000001, 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:36.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:36.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:36.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 76.484529566999996, 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:36.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:36.386 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:36.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:36.388 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:36.387+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:36.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:36.488 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:36.488+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:36.499 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:36.499 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:36.499 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:36.499 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:36.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:36.529 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:36.545 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:36.545 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:36.561 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:36.561 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:36.561 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:36.577 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:36.578 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:36.578 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:36.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:36.609 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:36.609 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:36.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:36.622 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:36.622+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:36.691 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:36.690+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:36.714 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:36.715 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:36.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:36.791 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:36.791+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:36.825 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:36.825 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:36.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:36.892 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:36.892+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:36.935 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:36.935 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:36.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:37.027 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:37.027+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:37.085 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.085 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:37.094 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:37.093+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:37.194 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:37.194+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:37.236 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.237 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:37.295 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:37.295+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:37.387 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.388 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:37.396 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:37.396+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:37.497 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.497 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.497 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:37.496+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:37.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:37.598 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:37.597+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:37.609 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.609 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.617 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:49:37.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:37.700 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:37.698+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:37.756 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.757 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:37.799 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:37.799+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:37.864 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.864 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.872 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:37.916 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:37.916+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:37.974 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:37.975 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:37.983 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:38.024 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:38.024+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:38.102 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:38.102+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:38.124 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:38.124 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:38.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:38.216 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:38.216+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:38.274 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:38.275 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:38.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:38.303 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:38.303+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:38.404 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:38.404+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:38.424 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:38.425 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:38.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:38.517 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:38.517+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:38.575 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:38.575 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:38.582 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:38.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:38.606 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:49:38.608 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:38.607+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:38.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:38.707+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:38.712 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:38.712 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:38.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:38.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:38.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:38.746 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:38.747 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:38.747 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:38.763 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:38.764 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:38.764 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:38.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:38.809 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:38.808+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:38.834 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:38.834 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:38.834 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:38.834 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:38.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:38.908 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:38.908+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:38.940 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:38.940 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:38.940 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:38.940 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:38.940 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:38.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:39.009 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:39.009+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:39.047 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.880488302000003, 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.873148967999995, 2024-09-17T07:49:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.872431894000002, 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.863204332999999, 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:39.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.853979765000005, 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:39.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.838505124999998, 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.071 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:39.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:39.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.829215614999995, 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:39.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:39.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.822625369999997, 2024-09-17T07:49:39.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:39.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:39.084 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.806442888999996, 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.085 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.797367969000007, 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.090 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.456929092999999, 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:39.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:39.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:39.100 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.454876815999995, 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:39.101 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:39.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:39.106 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.394645165, 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:39.107 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:39.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:39.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:39.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.233466015999994, 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:39.113 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:39.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.118 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.233441714999998, 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:39.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:39.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:39.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.233407338999996, 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:39.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:39.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.128 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.233385982000001, 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.129 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:39.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.230717338000005, 2024-09-17T07:49:39.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:39.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.220999765000002, 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.254889375999994, 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:39.151 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:39.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:39.153 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:39.151+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:39.253 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:39.253+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:39.259 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:39.260 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:39.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:39.354 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:39.354+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:39.410 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:39.411 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:39.411 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:39.411 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:39.411 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:39.419 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:39.462 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:39.462+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:39.522 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.355970389000007, 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:39.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.527 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.348310873000003, 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:39.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:39.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.339921305000004, 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:39.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:39.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.339550055000004, 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.539 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:39.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.331487089999996, 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:39.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:39.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:39.548 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.331115978, 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:39.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.321835573000001, 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:39.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:39.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:39.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.558 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.320176140000001, 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:39.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:39.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.314151315000004, 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.563 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:39.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:39.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.313900988, 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:39.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:39.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.306787150000005, 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:39.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.306270050999998, 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:39.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:39.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.287298883999995, 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:39.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:39.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.588 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.281042024000001, 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:39.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.274505962000006, 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.593 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:39.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:39.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:39.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.901639552000006, 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:39.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:39.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.899679144999993, 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:39.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.014010509000002, 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:39.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:39.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.616 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.703164870999998, 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:39.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:39.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.701803425999998, 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:39.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:39.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:39.628 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:39.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:39.630 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:39.629+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:39.730 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:39.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:39.730 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5] 2024-09-17T07:49:39.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-09-17T07:49:39.731 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:39.730+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:39.774 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:39.774 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:39.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:39.867 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:39.867+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:39.926 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:39.926 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:39.926 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:39.926 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:39.926 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:39.931 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:39.931+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:39.935 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:40.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-17T07:49:40.015+0000 7ffb533f1640 -1 mon.a@0(leader).osd e87 definitely_dead 0 2024-09-17T07:49:40.032 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:40.031+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:40.077 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.818978934, 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.450673318, 2024-09-17T07:49:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.758491062999994, 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.752634207, 2024-09-17T07:49:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.740941247999999, 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.672086664999995, 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.660187375999996, 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.609342828999999, 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.603334622000006, 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.597634264999996, 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.591894640000007, 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 87.258783592, 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.413682695000006, 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.407595971999996, 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.324139670999998, 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.317825239000001, 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.311453779000004, 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.305531302999995, 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.247548406999996, 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.185 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 80.289843735000005, 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:40.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:40.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:40.189 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:40.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:40.191 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:40.191+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:40.291 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:40.291+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:40.300 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:40.301 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:40.301 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:40.301 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:40.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:40.392 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:40.392+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:40.449 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:40.449 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:40.449 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:40.449 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:40.450 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:40.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:40.494 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:40.493+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.390394864000001, 2024-09-17T07:49:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.383055529999993, 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.382338455999999, 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.373110894999996, 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.363886327000003, 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.348411686999995, 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.339122176999993, 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.332531931999995, 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.316349450999994, 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.307274531000004, 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.966835654999997, 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:40.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:40.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.964783378000007, 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:40.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:40.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.904551726999998, 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:40.616 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:40.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:40.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:40.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.743372578000006, 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:40.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.743348276999996, 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:40.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:40.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:40.632 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.743313900999993, 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:40.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:40.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.743292543999999, 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:40.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:40.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.740623900000003, 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:40.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:40.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.730906327, 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:40.648 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:40.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:40.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.653 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.764795938000006, 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:40.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:40.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:40.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:40.659 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:40.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:49:40.661 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.5. 2024-09-17T07:49:40.662 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:40.660+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:40.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:40.761 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:40.761+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:40.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:40.812 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:40.813 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:40.813 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:40.813 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:40.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:40.862 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:40.862+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:40.876 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:40.892 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:40.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:40.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:40.909 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:40.909 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:40.924 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:40.927 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:40.928 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:40.928 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:40.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:40.977 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:40.977+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:41.064 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:41.064+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:41.078 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.078 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.087 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:41.170 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:41.170+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:41.228 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.228 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.237 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:41.266 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:41.266+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:41.367 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:41.366+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:41.377 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.377 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:41.468 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:41.468+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:41.529 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.529 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:41.569 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:41.568+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:49:41.648 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.648 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:41.701 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:41.701+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:41.758 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.759 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:41.770 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:41.770+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:41.871 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:41.871+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:41.905 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:41.905 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:41.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:41.972 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:41.972+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.016 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.016 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:42.073 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.074+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.126 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.126 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:42.174 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.175+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.236 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.237 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:42.288 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:42.288+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:42.346 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.346 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.355 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:42.376 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.376+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.477 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.477+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.496 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.496 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:42.578 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.579+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.605 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.606 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.614 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:42.679 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.679+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.721 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.721 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:42.780 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:42.780+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:42.868 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:42.901 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:42.902 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:42.903 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:42.902+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:42.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:43.003 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:43.002+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:43.040 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:43.040 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:43.048 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:43.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:43.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:43.064 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:43.080 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:43.081 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:43.081 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:43.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:43.103 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:43.104+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:43.152 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:43.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:43.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:43.152 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:43.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:43.205 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:43.204+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:43.261 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:43.261 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:43.261 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:43.261 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:43.261 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:43.269 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:43.305 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:43.306+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:43.407 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:43.407+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:43.412 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:43.412 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:43.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:43.412 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:43.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.245811003, 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:43.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.238151487, 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.417 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:43.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:43.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.229761919, 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.428 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.229390669, 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:43.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.221327704, 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:43.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:43.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:43.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.22095659199999, 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.438 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:43.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:43.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:43.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.21167618699999, 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:43.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:43.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.21001675399999, 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.448 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:43.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:43.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.203991929, 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.452 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:43.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.20374160199999, 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:43.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.196627764, 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.462 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.19611066500001, 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:43.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.177139498, 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.473 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.477 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.17088263799999, 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:43.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:43.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.164346576, 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.791480165999999, 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:43.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.789519759000001, 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.493 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:43.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:43.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 94.903851122999995, 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.499 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:43.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.504 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 90.593005485000006, 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:43.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 90.591644040000006, 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:43.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.516 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:43.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:43.518 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:43.517+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:43.618 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:43.618+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:43.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:49:43.661 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:43.661 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:43.661 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:43.661 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:43.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:43.719 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:43.719+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:43.814 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:43.814 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:43.814 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:43.814 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:43.814 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:43.819 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:43.819+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:43.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:43.920 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:43.919+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.707011662, 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.338706045999999, 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.646523791000007, 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.640666934999999, 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.628973975999997, 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.560119392999994, 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.548220103999995, 2024-09-17T07:49:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.497375556999998, 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.491367350000004, 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.485666992999995, 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.479927368000006, 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 91.146816319999999, 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.301715423000005, 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:44.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:44.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:44.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:44.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.295628699999995, 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.212172398999996, 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.205857967, 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:44.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:44.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.199486507000003, 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:44.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:44.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:44.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:44.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.193564030999994, 2024-09-17T07:49:44.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:44.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:44.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:44.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.135581134999995, 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:44.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:44.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:44.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 84.177876463000004, 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:44.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.080 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.081 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:44.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:44.083 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:44.082+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:44.183 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:44.183+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:44.192 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:44.192 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:44.192 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:44.192 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:44.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:44.285 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:44.284+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:44.298 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:44.298 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:44.299 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:44.299 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:44.299 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:44.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:44.385 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:44.385+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:44.406 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:44.406 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:44.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:44.406 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:44.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.23950531, 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:44.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.232165976, 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:44.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.231448902, 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:44.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.22222134099999, 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.421 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:44.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.212996773, 2024-09-17T07:49:44.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:44.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.19752213300001, 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:44.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.188232623, 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.18164237800001, 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.16545989700001, 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.156384977, 2024-09-17T07:49:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.815946100999994, 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.813893824000004, 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.753662172999995, 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.592483024000003, 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.592458722999993, 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.592424347000005, 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.592402989999997, 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.589734346, 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.580016772999997, 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.613906384000003, 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:44.508 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:44.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:44.510 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:44.508+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:44.610 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:44.609+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:44.617 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:44.617 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:44.617 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:44.617 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:44.626 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:49:44.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:44.722 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:44.722+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:44.780 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:44.780 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:44.780 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:44.780 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:44.780 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:44.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:44.829 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:44.829+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:44.912 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:44.912+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.765342203, 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.757682687, 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.749293119, 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:44.947 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.748921869, 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:44.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.74085890400001, 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.952 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:44.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:44.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.740487792, 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:44.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:44.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:44.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.731207387, 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:44.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:44.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.729547954, 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:44.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:44.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.723523129, 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:44.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:44.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.723272802, 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:44.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.716158964, 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.980 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.71564186499999, 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:44.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.69667069800001, 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.991 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:44.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:44.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.690413838, 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:44.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:44.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.683877776, 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:45.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:45.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:45.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.311011366000002, 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:45.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.011 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.309050959000004, 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:45.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:45.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.423382322999998, 2024-09-17T07:49:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.112536684999995, 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.111175239999994, 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:45.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:45.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:45.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.034 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:45.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:45.036 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:45.035+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:45.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:45.136 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:45.135+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:45.180 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:45.181 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:45.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:45.181 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:45.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:45.237 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:45.236+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:45.237 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:45.238 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:45.255 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:45.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:45.255 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:45.271 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:45.275 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:45.295 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:45.295 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:45.303 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:45.338 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:45.337+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:45.439 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:45.439+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:45.445 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:45.445 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:45.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:45.540 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:45.539+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:45.562 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:45.563 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:45.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:45.641 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:45.640+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:45.662 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:45.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:45.663 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T07:49:45.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:49:45.672 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:45.672 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:45.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:45.742 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:45.741+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:45.819 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:45.820 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:45.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:45.871 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:45.871+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:45.930 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:45.930 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:45.937 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:45.937 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:45.937 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 90, 2024-09-17T07:49:45.937 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:49:45.937 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:49:45.937 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:49:44.620813+0000", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:48:52.499074+0000", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:49:40.608922+0000", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 4, 2024-09-17T07:49:45.938 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:49:45.939 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:45.940 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:45.941 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:49:45.942 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:45.943 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:45.944 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:45.945 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:49:45.946 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:45.947 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:45.948 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:45.951 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:45.952 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "59", 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 6, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 59, 2024-09-17T07:49:45.953 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:45.954 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:45.955 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:45.956 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "new_default_pool27356", 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:48:02.539579+0000", 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:49:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:49:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "49", 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:49:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 49, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:49:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:49:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 29, 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31884 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/31884", 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/31884", 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/31884", 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/31884", 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:49:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 88, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:49:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:49:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:49:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 88, 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:49:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:49:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:49:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.3944549560546875, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31604 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/31604", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/31604", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/31604", 2024-09-17T07:49:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/31604", 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.456390380859375, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:49:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:49:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:49:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 46, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31606 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/31606", 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/31606", 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/31606", 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/31606", 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-09-17T07:49:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:49:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:49:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-17T07:49:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.2", 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:49:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:49:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [ 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "snaps": [ 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:46.007 INFO:teuthology.orchestra.run.smithi026.stdout: "begin": 2, 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "length": 1 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:49:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:49:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:49:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:46.009 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:46.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:46.012 INFO:tasks.thrashosds.thrasher:j is {'epoch': 90, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:49:44.620813+0000', 'last_up_change': '2024-09-17T07:48:52.499074+0000', 'last_in_change': '2024-09-17T07:49:40.608922+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 29, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 31884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 31884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 31884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 31884}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 31884}]}, 'public_addr': '172.21.15.26:6801/31884', 'cluster_addr': '172.21.15.26:6803/31884', 'heartbeat_back_addr': '172.21.15.26:6807/31884', 'heartbeat_front_addr': '172.21.15.26:6805/31884', 'state': ['exists']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 0, 'in': 1, 'weight': 0.3944549560546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 31604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 31604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 31604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 31604}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 31604}]}, 'public_addr': '172.21.15.94:6801/31604', 'cluster_addr': '172.21.15.94:6803/31604', 'heartbeat_back_addr': '172.21.15.94:6807/31604', 'heartbeat_front_addr': '172.21.15.94:6805/31604', 'state': ['exists']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 0.456390380859375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 39, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 31606}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 31606}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 31606}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 31606}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 31606}]}, 'public_addr': '172.21.15.94:6817/31606', 'cluster_addr': '172.21.15.94:6819/31606', 'heartbeat_back_addr': '172.21.15.94:6823/31606', 'heartbeat_front_addr': '172.21.15.94:6821/31606', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}, {'pgid': '3.1', 'osds': [3, 2]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '1.2', 'osds': [2, 4]}, {'pgid': '3.1', 'osds': [2, 4]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:49:46.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:49:46.012 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:46.010+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:46.111 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:46.110+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:46.154 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.155 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:46.212 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:46.211+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:46.265 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.266 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:46.315 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:46.315+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:46.324 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:49:46.325 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:49:46.338 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-17T07:49:45.444473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215549319,"num_objects":97,"num_object_clones":1,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":6,"num_read":3894,"num_read_kb":175713,"num_write":682,"num_write_kb":264995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":67109091,"num_keys_recovered":6,"num_objects_omap":12,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":411,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":693,"ondisk_log_size":693,"up":26,"acting":28,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1776184,"kb_used_data":260264,"kb_used_omap":8,"kb_used_meta":1515831,"kb_avail":470083016,"statfs":{"total":483183820800,"available":481365008384,"internally_reserved":0,"allocated":266510336,"data_stored":267102934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8520,"internal_metadata":1552211640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":41943210,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001454"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":26,"reported_epoch":67,"state":"stale+active+undersized","last_fresh":"2024-09-17T07:48:49.979339+0000","last_change":"2024-09-17T07:48:03.640213+0000","last_active":"2024-09-17T07:48:49.979339+0000","last_peered":"2024-09-17T07:48:49.979339+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640213+0000","last_became_peered":"2024-09-17T07:48:03.640213+0000","last_unstale":"2024-09-17T07:48:52.503858+0000","last_undegraded":"2024-09-17T07:48:49.979339+0000","last_fullsized":"2024-09-17T07:48:03.639399+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:21.591427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"45'7","reported_seq":80,"reported_epoch":48,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:05.372036+0000","last_change":"2024-09-17T07:48:01.879165+0000","last_active":"2024-09-17T07:48:05.372036+0000","last_peered":"2024-09-17T07:48:05.372036+0000","last_clean":"2024-09-17T07:47:41.593387+0000","last_became_active":"2024-09-17T07:47:45.602779+0000","last_became_peered":"2024-09-17T07:47:45.602779+0000","last_unstale":"2024-09-17T07:48:08.649038+0000","last_undegraded":"2024-09-17T07:48:01.878970+0000","last_fullsized":"2024-09-17T07:47:45.602289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:18.255015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":915,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:49:41.724688+0000","last_change":"2024-09-17T07:49:41.724688+0000","last_active":"2024-09-17T07:49:41.724688+0000","last_peered":"2024-09-17T07:49:41.724688+0000","last_clean":"2024-09-17T07:49:41.724688+0000","last_became_active":"2024-09-17T07:49:41.619856+0000","last_became_peered":"2024-09-17T07:49:41.619856+0000","last_unstale":"2024-09-17T07:49:41.724688+0000","last_undegraded":"2024-09-17T07:49:41.724688+0000","last_fullsized":"2024-09-17T07:49:41.724688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:07.730915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":756,"num_read_kb":27413,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12583028,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:49:40.615002+0000","last_change":"2024-09-17T07:48:05.649758+0000","last_active":"2024-09-17T07:49:40.615002+0000","last_peered":"2024-09-17T07:49:40.615002+0000","last_clean":"2024-09-17T07:49:40.615002+0000","last_became_active":"2024-09-17T07:48:03.642170+0000","last_became_peered":"2024-09-17T07:48:03.642170+0000","last_unstale":"2024-09-17T07:49:40.615002+0000","last_undegraded":"2024-09-17T07:49:40.615002+0000","last_fullsized":"2024-09-17T07:49:40.615002+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:30.026257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"54'4","reported_seq":79,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979236+0000","last_change":"2024-09-17T07:48:02.160661+0000","last_active":"2024-09-17T07:48:49.979236+0000","last_peered":"2024-09-17T07:48:49.979236+0000","last_clean":"2024-09-17T07:47:42.119185+0000","last_became_active":"2024-09-17T07:47:45.604354+0000","last_became_peered":"2024-09-17T07:47:45.604354+0000","last_unstale":"2024-09-17T07:48:52.503859+0000","last_undegraded":"2024-09-17T07:48:02.140633+0000","last_fullsized":"2024-09-17T07:47:45.603702+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:54:04.231757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"58'257","reported_seq":1196,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:40.614293+0000","last_change":"2024-09-17T07:48:01.916903+0000","last_active":"2024-09-17T07:49:40.614293+0000","last_peered":"2024-09-17T07:49:40.614293+0000","last_clean":"2024-09-17T07:47:43.970152+0000","last_became_active":"2024-09-17T07:47:45.604092+0000","last_became_peered":"2024-09-17T07:47:45.604092+0000","last_unstale":"2024-09-17T07:49:40.614293+0000","last_undegraded":"2024-09-17T07:48:01.915803+0000","last_fullsized":"2024-09-17T07:47:45.603408+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:28.677664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797533,"num_objects":17,"num_object_clones":1,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1339,"num_read_kb":77312,"num_write":306,"num_write_kb":58504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":60,"reported_epoch":88,"state":"active+undersized","last_fresh":"2024-09-17T07:49:40.614877+0000","last_change":"2024-09-17T07:48:53.510166+0000","last_active":"2024-09-17T07:49:40.614877+0000","last_peered":"2024-09-17T07:49:40.614877+0000","last_clean":"2024-09-17T07:48:50.494734+0000","last_became_active":"2024-09-17T07:48:53.510166+0000","last_became_peered":"2024-09-17T07:48:53.510166+0000","last_unstale":"2024-09-17T07:49:40.614877+0000","last_undegraded":"2024-09-17T07:49:40.614877+0000","last_fullsized":"2024-09-17T07:48:53.509462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:33.475636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"45'5","reported_seq":66,"reported_epoch":88,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:40.614417+0000","last_change":"2024-09-17T07:48:04.736433+0000","last_active":"2024-09-17T07:49:40.614417+0000","last_peered":"2024-09-17T07:49:40.614417+0000","last_clean":"2024-09-17T07:49:40.614417+0000","last_became_active":"2024-09-17T07:48:04.643512+0000","last_became_peered":"2024-09-17T07:48:04.643512+0000","last_unstale":"2024-09-17T07:49:40.614417+0000","last_undegraded":"2024-09-17T07:49:40.614417+0000","last_fullsized":"2024-09-17T07:49:40.614417+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:58.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'186","reported_seq":954,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:49:41.617247+0000","last_change":"2024-09-17T07:47:02.488118+0000","last_active":"2024-09-17T07:49:41.617247+0000","last_peered":"2024-09-17T07:49:41.617247+0000","last_clean":"2024-09-17T07:49:41.617247+0000","last_became_active":"2024-09-17T07:47:02.487807+0000","last_became_peered":"2024-09-17T07:47:02.487807+0000","last_unstale":"2024-09-17T07:49:41.617247+0000","last_undegraded":"2024-09-17T07:49:41.617247+0000","last_fullsized":"2024-09-17T07:49:41.617247+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:04:40.147948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777417,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1410,"num_read_kb":46187,"num_write":101,"num_write_kb":32778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+undersized","last_fresh":"2024-09-17T07:49:40.615065+0000","last_change":"2024-09-17T07:48:03.641341+0000","last_active":"2024-09-17T07:49:40.615065+0000","last_peered":"2024-09-17T07:49:40.615065+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.641341+0000","last_became_peered":"2024-09-17T07:48:03.641341+0000","last_unstale":"2024-09-17T07:49:40.615065+0000","last_undegraded":"2024-09-17T07:49:40.615065+0000","last_fullsized":"2024-09-17T07:48:03.640567+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:02.148375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":125,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:40.614855+0000","last_change":"2024-09-17T07:48:53.509681+0000","last_active":"2024-09-17T07:49:40.614855+0000","last_peered":"2024-09-17T07:49:40.614855+0000","last_clean":"2024-09-17T07:48:50.494470+0000","last_became_active":"2024-09-17T07:48:53.509681+0000","last_became_peered":"2024-09-17T07:48:53.509681+0000","last_unstale":"2024-09-17T07:49:40.614855+0000","last_undegraded":"2024-09-17T07:48:53.508978+0000","last_fullsized":"2024-09-17T07:48:53.508899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:56.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":143,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:40.614835+0000","last_change":"2024-09-17T07:48:09.653077+0000","last_active":"2024-09-17T07:49:40.614835+0000","last_peered":"2024-09-17T07:49:40.614835+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:48:09.653077+0000","last_became_peered":"2024-09-17T07:48:09.653077+0000","last_unstale":"2024-09-17T07:49:40.614835+0000","last_undegraded":"2024-09-17T07:48:09.652573+0000","last_fullsized":"2024-09-17T07:48:09.652495+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:35.420073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"55'78","reported_seq":226,"reported_epoch":88,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:49:40.614355+0000","last_change":"2024-09-17T07:48:04.691716+0000","last_active":"2024-09-17T07:49:40.614355+0000","last_peered":"2024-09-17T07:49:40.614355+0000","last_clean":"2024-09-17T07:49:40.614355+0000","last_became_active":"2024-09-17T07:48:04.643230+0000","last_became_peered":"2024-09-17T07:48:04.643230+0000","last_unstale":"2024-09-17T07:49:40.614355+0000","last_undegraded":"2024-09-17T07:49:40.614355+0000","last_fullsized":"2024-09-17T07:49:40.614355+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:16:12.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":88,"num_write_kb":24609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":27,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979315+0000","last_change":"2024-09-17T07:48:26.474017+0000","last_active":"2024-09-17T07:48:49.979315+0000","last_peered":"2024-09-17T07:48:49.979315+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640668+0000","last_became_peered":"2024-09-17T07:48:03.640668+0000","last_unstale":"2024-09-17T07:48:52.503860+0000","last_undegraded":"2024-09-17T07:48:25.341555+0000","last_fullsized":"2024-09-17T07:48:03.639432+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:44.698558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":140,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:49:41.673861+0000","last_change":"2024-09-17T07:49:41.673861+0000","last_active":"2024-09-17T07:49:41.673861+0000","last_peered":"2024-09-17T07:49:41.673861+0000","last_clean":"2024-09-17T07:49:41.673861+0000","last_became_active":"2024-09-17T07:49:41.619674+0000","last_became_peered":"2024-09-17T07:49:41.619674+0000","last_unstale":"2024-09-17T07:49:41.673861+0000","last_undegraded":"2024-09-17T07:49:41.673861+0000","last_fullsized":"2024-09-17T07:49:41.673861+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.123670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":38,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979198+0000","last_change":"2024-09-17T07:48:25.422858+0000","last_active":"2024-09-17T07:48:49.979198+0000","last_peered":"2024-09-17T07:48:49.979198+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640271+0000","last_became_peered":"2024-09-17T07:48:03.640271+0000","last_unstale":"2024-09-17T07:48:52.503861+0000","last_undegraded":"2024-09-17T07:48:25.340395+0000","last_fullsized":"2024-09-17T07:48:03.639357+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:03.926215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":78,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:40.614899+0000","last_change":"2024-09-17T07:48:01.981238+0000","last_active":"2024-09-17T07:49:40.614899+0000","last_peered":"2024-09-17T07:49:40.614899+0000","last_clean":"2024-09-17T07:47:41.593808+0000","last_became_active":"2024-09-17T07:47:45.603610+0000","last_became_peered":"2024-09-17T07:47:45.603610+0000","last_unstale":"2024-09-17T07:49:40.614899+0000","last_undegraded":"2024-09-17T07:48:01.980889+0000","last_fullsized":"2024-09-17T07:47:45.603119+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:18:17.830983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":30,"reported_epoch":67,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:49.979225+0000","last_change":"2024-09-17T07:48:04.643889+0000","last_active":"2024-09-17T07:48:49.979225+0000","last_peered":"2024-09-17T07:48:49.979225+0000","last_clean":"2024-09-17T07:48:02.630800+0000","last_became_active":"2024-09-17T07:48:03.640398+0000","last_became_peered":"2024-09-17T07:48:03.640398+0000","last_unstale":"2024-09-17T07:48:52.503862+0000","last_undegraded":"2024-09-17T07:48:03.642853+0000","last_fullsized":"2024-09-17T07:48:03.639625+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:08.008675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":70,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:40.614283+0000","last_change":"2024-09-17T07:48:01.880716+0000","last_active":"2024-09-17T07:49:40.614283+0000","last_peered":"2024-09-17T07:49:40.614283+0000","last_clean":"2024-09-17T07:46:51.988421+0000","last_became_active":"2024-09-17T07:47:45.604099+0000","last_became_peered":"2024-09-17T07:47:45.604099+0000","last_unstale":"2024-09-17T07:49:40.614283+0000","last_undegraded":"2024-09-17T07:48:01.880450+0000","last_fullsized":"2024-09-17T07:47:45.603421+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:28.433406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":66,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:49:40.614734+0000","last_change":"2024-09-17T07:48:53.509737+0000","last_active":"2024-09-17T07:49:40.614734+0000","last_peered":"2024-09-17T07:49:40.614734+0000","last_clean":"2024-09-17T07:48:50.494367+0000","last_became_active":"2024-09-17T07:48:53.509737+0000","last_became_peered":"2024-09-17T07:48:53.509737+0000","last_unstale":"2024-09-17T07:49:40.614734+0000","last_undegraded":"2024-09-17T07:48:53.508926+0000","last_fullsized":"2024-09-17T07:48:53.508829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"45'5","reported_seq":98,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:49:41.666630+0000","last_change":"2024-09-17T07:49:41.666630+0000","last_active":"2024-09-17T07:49:41.666630+0000","last_peered":"2024-09-17T07:49:41.666630+0000","last_clean":"2024-09-17T07:49:41.666630+0000","last_became_active":"2024-09-17T07:49:41.619584+0000","last_became_peered":"2024-09-17T07:49:41.619584+0000","last_unstale":"2024-09-17T07:49:41.666630+0000","last_undegraded":"2024-09-17T07:49:41.666630+0000","last_fullsized":"2024-09-17T07:49:41.666630+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:17.795246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":89,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":3},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80740966,"num_objects":44,"num_object_clones":1,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3719,"num_read_kb":175578,"num_write":531,"num_write_kb":132277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":25165978,"num_keys_recovered":3,"num_objects_omap":5,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":411,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135344128,"data_stored":138413155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":566,"ondisk_log_size":566,"up":6,"acting":7,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":1,"acting":1,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217915,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":44,"num_write_kb":131084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":41943113,"num_keys_recovered":3,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":243322880,"data_stored":243270003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":10,"acting":11,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640293,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377388,"kb_used_data":71708,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93994452,"statfs":{"total":96636764160,"available":96250318848,"internally_reserved":0,"allocated":73428992,"data_stored":75931021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.625,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.433}]},{"osd":2,"last update":"Tue Sep 17 07:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40600000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.254}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.316},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.32800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319040,"kb_used_data":17516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94052800,"statfs":{"total":96636764160,"available":96310067200,"internally_reserved":0,"allocated":17936384,"data_stored":17586425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.309}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.24299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355816,"kb_used_data":54296,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94016024,"statfs":{"total":96636764160,"available":96272408576,"internally_reserved":0,"allocated":55599104,"data_stored":54959535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.315,"15min":0.32700000000000001},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.314,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.246},{"interface":"front","average":{"1min":0.247,"5min":0.30299999999999999,"15min":0.312},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.35299999999999998,"5min":0.434,"15min":0.434},"last":0.22900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.628,"15min":0.628},"last":0.318},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.45500000000000002,"15min":0.46899999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.49399999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.33000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438}]}]},{"osd":1,"up_from":11,"seq":47244640294,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422012,"kb_used_data":116332,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93949828,"statfs":{"total":96636764160,"available":96204623872,"internally_reserved":0,"allocated":119123968,"data_stored":118470019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.441,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.35499999999999998,"15min":0.36099999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.498,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.29699999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61299999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.58099999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.374}]}]},{"osd":0,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301928,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069912,"statfs":{"total":96636764160,"available":96327589888,"internally_reserved":0,"allocated":421888,"data_stored":155934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88092672,"data_stored":88080476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16777216,"data_stored":16777216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75517952,"data_stored":75497640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29376512,"data_stored":29360445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29376512,"data_stored":29360405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":64000000,"data_stored":67109277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:49:46.340 INFO:tasks.thrashosds.thrasher:Setting 4.0 to [4, 5] 2024-09-17T07:49:46.340 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.0', '4', '5'] 2024-09-17T07:49:46.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.0 4 5 2024-09-17T07:49:46.418 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.418 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.418 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:46.418+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:46.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:46.515 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:46.514+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:46.570 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.570 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:46.615 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:46.615+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:46.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:49:46.716 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:46.715+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:46.722 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.722 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.731 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:46.817 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:46.817+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:46.835 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.835 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:46.919 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:46.917+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:46.941 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:46.941 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:46.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:47.020 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:47.020+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:47.087 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:47.087 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:47.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:47.139 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:47.139+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:47.197 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:47.198 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:47.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:47.247 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:47.247+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:47.276 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:47.333 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:47.333+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:47.349 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:47.349 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:47.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:47.391 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:47.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:47.407 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:47.423 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:47.422+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:47.424 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:47.424 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:47.425 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:47.440 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:47.441 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:47.441 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:47.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:47.496 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.496 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.329396348, 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.497 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:47.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:47.501 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.321736832, 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:47.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:47.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.313347264, 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:47.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:47.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:47.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.312976014, 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.513 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:47.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:47.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.30491304900001, 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:47.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.521 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.304541937, 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:47.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.295261532, 2024-09-17T07:49:47.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:47.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.293602099, 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:47.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:47.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:47.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.536 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.287577274, 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:47.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:47.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.287326947, 2024-09-17T07:49:47.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:47.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:47.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.280213109, 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:47.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:47.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.27969600999999, 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:47.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.26072484300001, 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.558 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:47.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:47.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.254467983, 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:47.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:47.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:47.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.247931921, 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:47.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:47.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.875065511000003, 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:47.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:47.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:47.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.873105104000004, 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:47.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:47.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 98.987436467999999, 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:47.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:47.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:47.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:47.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.592 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 94.676590829999995, 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:47.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:47.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:47.598 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 94.675229384999994, 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:47.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:47.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:47.604 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:47.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:47.606 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:47.605+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:47.631 INFO:teuthology.orchestra.run.smithi026.stderr:set 4.0 pg_upmap mapping to [4,5] 2024-09-17T07:49:47.632 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:49:47.706 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:47.706+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:47.734 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:47.735 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:47.735 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:47.735 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:47.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:47.827 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:47.827+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:47.885 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:47.885 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:47.885 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:47.885 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:47.885 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:47.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:47.907 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:47.907+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:48.008 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:48.008+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:48.041 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:48.041 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:48.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:48.041 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:48.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.779188039, 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:48.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 100.410882423, 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:48.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.718700167999998, 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:48.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:48.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.712843312000004, 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:48.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.701150353000003, 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:48.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.632295769999999, 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:48.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.620396481, 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:48.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.569551934000003, 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:48.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:48.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:48.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.563543726999995, 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.087 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:48.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:48.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.557843370000001, 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:48.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:48.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:48.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.099 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.552103744999997, 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:48.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 95.218992697000004, 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.106 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:48.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:48.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.373891799999996, 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:48.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:48.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.367805077, 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:48.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:48.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.284348776000002, 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:48.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:48.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:48.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.130 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.278034344000005, 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:48.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:48.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:48.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.136 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.271662883999994, 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:48.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:48.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:48.142 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.265740407999999, 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:48.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.148 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.207757512000001, 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:48.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:48.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:48.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:48.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 88.250052839999995, 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:48.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.159 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:48.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:48.161 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:48.160+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:48.261 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:48.261+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:48.270 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:48.270 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:48.270 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:48.270 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:48.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:48.362 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:48.362+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:48.377 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:48.377 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:48.377 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:48.377 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:48.377 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:48.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:48.464 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:48.463+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:48.484 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.317674622, 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.310335288, 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.309618214, 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.30039065299999, 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.291166085, 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.27569144500001, 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.266401935, 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.25981169000001, 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.24362920900001, 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.234554289, 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.89411541299999, 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.892063136, 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.831831485, 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.670652336000003, 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.670628034999993, 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.670593659000005, 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.670572301999997, 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.667903658, 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.658186084999997, 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 88.692075696000003, 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:48.586 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:48.588 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:48.587+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:48.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:48.688 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:48.688+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:48.700 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:48.700 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:48.701 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:48.701 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:48.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:48.789 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:48.788+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:48.852 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:48.852 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:48.852 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:48.852 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:48.853 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:48.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:48.890 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:48.889+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.798037235, 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:48.965 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:48.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:48.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.79037771900001, 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.970 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:48.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.975 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.78198815099999, 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.78161690100001, 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.773553936, 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.773182824, 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.763902419, 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:48.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:48.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.762242986, 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:49.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.75621816100001, 2024-09-17T07:49:49.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:49.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:49.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.755967834, 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:49.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:49.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.74885399599999, 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:49.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:49.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.748336897, 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:49.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:49.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.72936573, 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.026 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:49.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.72310887, 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:49.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.716572808, 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:49.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:49.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:49.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.34370639799999, 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:49.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:49.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:49.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.341745991, 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:49.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:49.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 100.45607735500001, 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:49.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:49.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:49.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.060 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.145231717000001, 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:49.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.066 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.143870272000001, 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:49.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:49.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.072 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:49.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:49.074 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:49.073+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:49.174 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:49.174+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:49.217 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:49.218 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:49.218 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:49.218 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:49.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:49.311 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:49.311+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:49.369 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:49.369 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:49.369 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:49.369 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:49.369 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:49.376 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:49.376+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:49.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:49.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:49.504 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:49.504+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:49.523 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:49.523 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:49.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:49.523 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:49.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 107.264446651, 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:49.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:49.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.896141035, 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.529 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:49.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:49.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.20395877999999, 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:49.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:49.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.198101924, 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:49.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:49.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:49.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.186408965, 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.547 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:49.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:49.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.11755438199999, 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.553 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:49.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:49.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.105655093, 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:49.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.054810546, 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:49.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:49.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.04880233900001, 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:49.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:49.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:49.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:49.578 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.043101982, 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:49.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:49.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.03736235700001, 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:49.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:49.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:49.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.704251309, 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.593 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:49.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.859150412000005, 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:49.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:49.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.853063688999995, 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:49.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.612 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.769607387999997, 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:49.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.618 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.763292956000001, 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.756921496000004, 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.750999019999995, 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.693016123999996, 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.735311452000005, 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:49.649 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:49.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:49:49.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:49.651 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:49.652 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:49.652 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:49.653 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:49.653 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:49.654 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:49.654 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:49.656 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:49.656 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:49.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:49.657 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:49.650+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:49.751 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:49.750+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:49.767 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:49.767 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:49.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:49.852 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:49.850+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:49.875 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:49.875 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:49.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:49.969 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:49.968+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:50.023 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.023 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.031 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:50.075 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:50.075+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:50.133 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.133 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.141 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:50.182 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:50.182+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:50.256 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:50.256+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:50.283 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.283 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:50.377 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:50.377+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:50.435 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.435 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:50.485 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:50.484+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:50.559 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:50.558+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:50.585 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.585 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:49:50.660 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:50.658+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:50.697 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.697 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.706 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:50.760 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:50.760+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:50.807 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.807 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:50.861 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:50.860+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:50.923 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:50.923 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:50.932 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:50.977 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:50.976+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:51.030 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:51.030 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:51.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:51.062 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:51.062+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:51.164 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:51.163+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:51.175 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:51.175 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:51.184 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:51.265 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:51.264+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:51.290 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:51.291 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:51.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:51.366 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:51.365+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:51.401 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:51.401 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:51.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:51.493 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:51.493+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:51.551 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:51.552 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:51.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:51.602 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:51.601+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:51.657 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:51.701 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:51.701+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:51.705 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:51.706 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:51.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:51.770 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:51.769+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:51.776 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:51.776 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:51.792 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:51.793 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:51.793 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:51.808 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:51.808 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:51.809 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:51.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:51.856 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:51.857 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:51.857 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:51.857 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:51.857 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:51.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:51.909 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:51.909+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:51.968 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:51.968 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:51.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:51.968 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:51.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.968 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.70926489199999, 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:51.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:51.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:51.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.973 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 104.34095927600001, 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:51.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:51.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.978 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.648777021, 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:51.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:51.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:51.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.64292016500001, 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:51.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:51.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:51.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.63122720600001, 2024-09-17T07:49:51.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:51.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:51.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:51.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.562372623, 2024-09-17T07:49:51.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:51.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:52.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:52.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.550473334, 2024-09-17T07:49:52.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:52.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:52.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.49962878700001, 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:52.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:52.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:52.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.49362058, 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:52.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:52.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:52.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.487920223, 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:52.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:52.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.482180598, 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:52.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 99.149069549999993, 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:52.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:52.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.303968652999998, 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.297881930000003, 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.214425629000004, 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.208111196999994, 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.201739736999997, 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.195817261000002, 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 96.137834365000003, 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 92.180129692999998, 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.086 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:52.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:52.088 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:52.086+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:52.187 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:52.187 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:52.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:52.187 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:52.188 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:52.186+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:52.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:52.288 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:52.288+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:52.333 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:52.333 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:52.333 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:52.333 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:52.333 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:52.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:52.389 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:52.389+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.27623334, 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:52.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.268894006, 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:52.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.268176932, 2024-09-17T07:49:52.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.258949371, 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:52.457 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:52.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.24972480300001, 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.462 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:52.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.234250163, 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:52.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.224960653, 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.475 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.218370408, 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.476 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:52.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:52.480 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.202187927, 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:52.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:52.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.19311300699999, 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.852674131, 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:52.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:52.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:52.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:52.497 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.850621854, 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:52.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:52.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:52.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.790390203, 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:52.503 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:52.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:52.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.629211053999995, 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:52.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:52.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:52.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.629186752999999, 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:52.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.629152376999997, 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.629131020000003, 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.626462376000006, 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.616744803000003, 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.650634413999995, 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:52.550 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:52.551 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:52.550+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:52.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:52.648 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [0, 3, 5] live_osds: [1, 2, 4] 2024-09-17T07:49:52.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:49:52.648 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-17T07:49:52.648 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T07:49:52.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T07:49:52.651 INFO:tasks.ceph.osd.5:Started 2024-09-17T07:49:52.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:49:52.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:49:52.652 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:52.651+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:52.667 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:52.667 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:52.667 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:52.667 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:52.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:52.746 INFO:teuthology.orchestra.run.smithi094.stderr:no valid command found; 10 closest matches: 2024-09-17T07:49:52.746 INFO:teuthology.orchestra.run.smithi094.stderr:0 2024-09-17T07:49:52.746 INFO:teuthology.orchestra.run.smithi094.stderr:1 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:2 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:abort 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:assert 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:config diff 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:config diff get 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:config get 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:config help [] 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:config set ... 2024-09-17T07:49:52.747 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: invalid command 2024-09-17T07:49:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:49:52.749 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-17T07:49:52.760 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:52.760+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:52.821 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:52.821 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:52.821 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:52.821 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:52.821 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:52.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:52.871 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:52.871+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:52.957 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:52.956+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:52.974 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:52.974 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.808096252, 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.975 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:52.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:52.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.80043673599999, 2024-09-17T07:49:52.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:52.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:52.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.792047168, 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:52.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.791675918, 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:52.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:52.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:52.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.783612953, 2024-09-17T07:49:52.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:52.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:52.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:52.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.78324184100001, 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.77396143599999, 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.004 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:53.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:53.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.77230200299999, 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:53.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:53.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.766277178, 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:53.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:53.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.76602685100001, 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:53.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:53.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.758913013, 2024-09-17T07:49:53.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:53.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:53.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:53.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.758395914, 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:53.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:53.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:53.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.739424747, 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:53.034 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:53.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:53.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:53.038 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.73316788699999, 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.726631825, 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:53.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.353765415, 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:53.048 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:53.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:53.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:53.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.351805008, 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:53.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 104.46613637199999, 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 100.155290734, 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 100.153929289, 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.077 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:53.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:53.079 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:53.078+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:53.179 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:53.179+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:53.222 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:53.222 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:53.222 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:53.222 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:53.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:53.233 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:49:53.231+0000 7f6b72011440 -1 Falling back to public interface 2024-09-17T07:49:53.314 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:53.314+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:53.372 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:53.372 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:53.372 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:53.372 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:53.372 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:53.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:53.421 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:53.421+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:53.482 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:53.482+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 111.265286553, 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.896980937, 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.204798682, 2024-09-17T07:49:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.198941826, 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.18724886699999, 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.118394284, 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.10649499500001, 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.05565044799999, 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.049642241, 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.04394188400001, 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.038202259, 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 100.705091211, 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.859990314000001, 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:53.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.853903591000005, 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.770447290000007, 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.764132857999996, 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.757761398, 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.751838922000005, 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.693856026000006, 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 93.736151354, 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:53.652 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:53.654 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:53.654 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:49:53.655 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:53.652+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:53.754 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:53.753+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:53.758 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:53.758 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:53.758 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:53.758 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:53.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:53.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:53.857 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:53.857+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:53.905 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:53.906 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:53.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:53.915 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:53.930 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:53.946 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:53.946 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:53.946 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:53.956 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:53.955+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:53.962 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:53.962 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:53.963 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:53.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:54.012 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.013 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:54.057 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:54.056+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:54.123 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.124 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:54.175 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:54.175+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:54.233 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.233 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.241 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:54.258 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:54.258+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:54.359 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:54.359+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:54.382 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.382 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.391 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:54.460 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:54.459+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:54.533 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.533 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:54.585 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:54.585+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:54.644 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.644 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:54.695 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:54.694+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:54.763 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:54.763+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:54.796 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.797 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:54.864 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:54.864+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:54.906 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:54.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:54.965 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:54.965+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:55.011 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.011 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.020 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:55.066 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:55.066+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:55.170 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:55.170+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:55.233 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.233 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:55.283 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:55.282+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:55.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:55.368+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:55.382 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.383 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.391 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:55.471 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:55.470+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:55.492 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.493 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:55.571 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:55.571+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:55.602 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.602 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:55.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:49:55.694 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:55.694+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:55.753 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.753 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:55.805 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:55.805+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:55.869 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.869 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:55.874 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:55.874+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:55.878 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:55.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:49:55.989 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:55.989 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:56.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:56.011 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:56.011+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:56.068 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:56.076 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:56.075+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:56.083 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:56.083 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:56.099 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:56.100 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:56.100 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:56.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:56.114 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:56.115 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:56.115 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:56.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:56.194 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:56.193+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:56.247 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.081271132, 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:56.248 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:56.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.073931798, 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:56.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.257 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.073214724, 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:56.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.06398716299999, 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.054762595, 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.03928795500001, 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.029998445, 2024-09-17T07:49:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.02340820000001, 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.007225719, 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.998150799, 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:56.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.294 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 108.65771192299999, 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.295 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:56.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.300 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 108.655659646, 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:56.301 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:56.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:56.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 108.59542799499999, 2024-09-17T07:49:56.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:56.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.313 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.434248846, 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.318 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.43422454500001, 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:56.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:56.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:56.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:56.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:56.324 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.434190169, 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.434168812, 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:56.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:56.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.431500168, 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:56.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:56.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.421782595, 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:56.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:56.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:56.345 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.455672206000003, 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:56.346 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:56.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:56.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:56.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:56.352 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:56.354 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:56.353+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:56.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:56.455 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:56.454+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:56.457 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:56.457 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:56.457 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:56.457 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:56.466 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:56.556 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:56.555+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:56.567 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:56.567 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:56.567 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:56.567 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:56.567 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:56.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:49:56.656 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:56.655+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:56.681 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.51506139999999, 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.507401884, 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.49901231600001, 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.498641066, 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.490578101, 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.490206989, 2024-09-17T07:49:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.480926584, 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.479267151, 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:56.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.723 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.473242326, 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:56.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:56.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.472991999, 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:56.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:56.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.46587816100001, 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:56.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:56.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.739 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.465361062, 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:56.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.446389895, 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:56.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.440133035, 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.750 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.43359697299999, 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:56.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:56.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.06073056300001, 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.760 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:56.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:56.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:56.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.05877015599999, 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:56.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.17310152, 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.862255882, 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 103.860894437, 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:56.790 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:56.791 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete...2024-09-17T07:49:56.723+0000 7f057964b640 0 -- 172.21.15.98:0/2998672302 >> [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] conn(0x7f0554008980 msgr2=0x7f055400ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:49:56.791 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:49:56.691+0000 7f6b72011440 -1 osd.5 68 log_to_monitors true 2024-09-17T07:49:56.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:56.793 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:56.791+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:56.892 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:56.892+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:56.935 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:56.935 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:56.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:57.028 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:57.028+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:57.085 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:57.086 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:57.086 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:57.086 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:57.086 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:57.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:57.094 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:57.093+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:57.195 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:57.194+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:57.237 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.97880213800001, 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:49:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:49:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.61049652200001, 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:49:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.918314267, 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:49:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:49:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:49:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:49:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:49:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.91245741100001, 2024-09-17T07:49:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:49:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:49:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:49:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.900764452, 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.831909869, 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.82001058, 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:49:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:49:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.769166033, 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:49:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.763157826, 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:49:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.757457469, 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:49:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:49:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.751717844, 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 104.41860679600001, 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:49:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.573505899, 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:49:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:49:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:49:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:49:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.567419176, 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:49:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:49:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.483962875, 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:49:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:49:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.47764844300001, 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:49:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:49:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.471276983, 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:49:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.465354507, 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 101.407371611, 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:49:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:49:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:49:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 97.449666938999997, 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:49:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:57.357 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:57.359 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:57.359 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:57.358+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:57.460 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:57.459+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:57.460 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:57.460 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:57.460 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:57.461 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:57.469 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:57.561 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:57.560+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:57.606 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:57.606 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:57.614 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:57.662 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:57.661+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:57.674 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:49:57.672+0000 7f6b657e8640 -1 osd.5 95 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.55225606099999, 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:49:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.544916727, 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:49:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.54419965300001, 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:49:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.534972092, 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.525747524, 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:49:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.510272884, 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:49:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.500983374, 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:49:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:49:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:49:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.494393129, 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:49:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.478210648, 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:49:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:49:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.469135728, 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.128696852, 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.126644575, 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:49:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:49:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.06641292400001, 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:49:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:49:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.905233775, 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:49:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.905209474, 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.787 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:49:57.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:49:57.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.791 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.905175098, 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:49:57.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:49:57.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:49:57.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.90515374100001, 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:49:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:49:57.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:49:57.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.902485097, 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.89276752400001, 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:49:57.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:49:57.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 97.926657134999999, 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:49:57.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:49:57.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:49:57.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:49:57.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:49:57.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:49:57.818 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:57.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:49:57.820 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:57.818+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:57.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:57.920 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:57.920+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:57.927 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:57.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:57.927 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:49:57.927 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:57.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:57.957 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:57.957 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:57.957 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:57.958 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:58.022 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:58.021+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:58.077 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:58.078 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:49:58.078 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:49:58.078 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:49:58.078 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:58.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:58.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:49:58.127 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:58.127+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:58.224 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:58.223+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:58.228 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.066296712, 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:49:58.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.05863719600001, 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:58.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:49:58.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.05024762799999, 2024-09-17T07:49:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:49:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:49:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:49:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.04987637799999, 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:49:58.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:49:58.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.041813413, 2024-09-17T07:49:58.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:49:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.041442301, 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:49:58.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.03216189600001, 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:49:58.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:49:58.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.030502463, 2024-09-17T07:49:58.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:49:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.02447763799999, 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:49:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:49:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:49:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:49:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.024227311, 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:49:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:49:58.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.01711347299999, 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.284 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:49:58.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:49:58.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:49:58.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:49:58.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:58.289 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.016596374, 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:49:58.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:49:58.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 115.997625207, 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:49:58.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 115.99136834700001, 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:49:58.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 115.984832285, 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:49:58.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.611965875, 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:49:58.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:49:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:49:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.315 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.610005468, 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:49:58.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.72433683200001, 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:49:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.413491194, 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:49:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:49:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:49:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.412129749, 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:49:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:49:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:49:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:49:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:49:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:49:58.340 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:49:58.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:58.342 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:49:58.342 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:49:58.342 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:49:58.343 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:49:58.343 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:49:58.343 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:49:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:49:58.344 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:49:58.345 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:49:58.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:49:58.345 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:58.340+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:58.442 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:58.442+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:58.486 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:58.486 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:58.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:49:58.543 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:49:58.542+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:49:58.637 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:58.637 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:58.643 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:58.643+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:58.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:49:58.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:49:58.744 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:58.744+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:58.750 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:58.750 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:58.760 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:49:58.845 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:58.844+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:58.862 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:58.862 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:58.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:49:58.945 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:58.944+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:58.968 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:58.968 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:58.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:49:59.046 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:59.046+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:59.114 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:59.114 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:59.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:49:59.166 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:59.165+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:59.220 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:59.220 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:59.220 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:49:59.220 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:59.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:49:59.268 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:59.267+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:59.349 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:59.349+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:59.365 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:59.365 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:49:59.365 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:49:59.365 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:49:59.365 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:59.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:49:59.450 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:59.449+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:49:59.471 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:49:59.471 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:49:59.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:49:59.472 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-09-17T07:49:59.472 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:49:59.480 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:49:59.552 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:59.551+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:59.588 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:59.589 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:59.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:49:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:49:59.665 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:59.664+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:59.744 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:59.744 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:59.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:49:59.754 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:49:59.753+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:49:59.854 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:49:59.854 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:49:59.855 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:49:59.854+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:49:59.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:49:59.955 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:49:59.955+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:00.005 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:00.005 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:00.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:00.057 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:00.057+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:00.117 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:00.117 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:00.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:00.157 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:00.157+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:00.257 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:00.257+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:00.268 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:00.268 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:00.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:00.346 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:00.358 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:00.357+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:00.380 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:00.381 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:00.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:00.446 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:00.459 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:00.459+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:00.462 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:00.462 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:00.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:00.493 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:00.509 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-17T07:50:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:00.513 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:00.529 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:00.529 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:00.529 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:00.529 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:00.529 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:00.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:00.561 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:00.560+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:00.637 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.470996591, 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.638 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:00.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.46365725699999, 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:00.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.462940183, 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:00.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:00.651 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.453712622, 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:00.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.444488054, 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:00.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.429013414, 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:00.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.41972390399999, 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:00.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.413133659, 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.669 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:00.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:00.673 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.39695117799999, 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:00.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.677 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.38787625800001, 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:00.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:00.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.047437382, 2024-09-17T07:50:00.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:00.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.04538510499999, 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:00.688 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:00.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.985153454, 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:00.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:00.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:00.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.82397430499999, 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:00.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:00.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.823950004, 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:00.705 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:00.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:00.710 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.82391562799999, 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:00.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:00.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:00.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.823894271, 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:00.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:00.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.720 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.821225627, 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:00.721 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:00.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:00.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.811508054, 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:00.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.84539766499999, 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:00.731 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:00.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:00.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:00.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:00.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:00.738 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:00.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:00.740 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:00.739+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:00.840 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:00.839+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:00.879 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:00.887 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:00.971 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:00.971+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:01.025 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:01.025 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:01.025 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:01.025 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:01.025 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:01.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:01.074 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:01.074+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:01.142 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:01.142+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:01.171 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:774f7988:ns::rbd_directory:head, version: 45'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 1, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:774f7988:ns::rbd_directory:head@45'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.667301+0000", 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.50300964299999995, 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025967999999999998, 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667301+0000", 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667301+0000", 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667302+0000", 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5000000000000003e-07 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667311+0000", 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4199999999999996e-06 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667322+0000", 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0984e-05 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667332+0000", 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.9939999999999995e-06 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667373+0000", 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0878999999999999e-05 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667380+0000", 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5919999999999997e-06 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.667561+0000", 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000181261 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.50195405000000004, 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:01.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.49894117700000001, 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:01.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.49667425199999998, 2024-09-17T07:50:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:01.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.494504634, 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:01.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.193 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.49121526700000001, 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.197 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.488322009, 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.198 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.48570913799999998, 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:01.202 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:01.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.48287085299999999, 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:01.206 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.690315+0000", 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.47999581299999999, 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024999700000000001, 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690315+0000", 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690315+0000", 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690316+0000", 2024-09-17T07:50:01.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690324+0000", 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2160000000000006e-06 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690332+0000", 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9419999999999995e-06 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690346+0000", 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4256999999999999e-05 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690373+0000", 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7535e-05 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.690565+0000", 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000191425 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.213 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.47872701400000001, 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:01.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.217 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.695629+0000", 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.47468150999999997, 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093849299999999999, 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695629+0000", 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695629+0000", 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695630+0000", 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695646+0000", 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5366000000000002e-05 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695666+0000", 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0937e-05 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695679+0000", 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2316e-05 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.695783+0000", 2024-09-17T07:50:01.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010426500000000001 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.696568+0000", 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078463899999999995 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.699080+0000", 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.47123027899999997, 2024-09-17T07:50:01.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00078029399999999995, 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699080+0000", 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699080+0000", 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699081+0000", 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.46e-07 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699086+0000", 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699093+0000", 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0840000000000003e-06 2024-09-17T07:50:01.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699097+0000", 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9670000000000003e-06 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699148+0000", 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0890999999999998e-05 2024-09-17T07:50:01.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.699861+0000", 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071299099999999995 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.702284+0000", 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.46802624399999998, 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075438600000000001, 2024-09-17T07:50:01.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702284+0000", 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702284+0000", 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702285+0000", 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8400000000000004e-07 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702292+0000", 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2609999999999996e-06 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702302+0000", 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1730000000000001e-06 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702306+0000", 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.87e-06 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.702347+0000", 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0198999999999999e-05 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.703039+0000", 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069229899999999995 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.46449215799999999, 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.232 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.709348+0000", 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.460962714, 2024-09-17T07:50:01.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00077391000000000005, 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709348+0000", 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709348+0000", 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709348+0000", 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6199999999999999e-07 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709354+0000", 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8359999999999997e-06 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709361+0000", 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1060000000000002e-06 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709365+0000", 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9670000000000003e-06 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.709419+0000", 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3653999999999998e-05 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.710122+0000", 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00070298500000000005 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.712657+0000", 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.457653802, 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00073705199999999998, 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712657+0000", 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712657+0000", 2024-09-17T07:50:01.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712657+0000", 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5100000000000001e-07 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712662+0000", 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7620000000000003e-06 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712669+0000", 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0280000000000003e-06 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712673+0000", 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.963e-06 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.712697+0000", 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3609e-05 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.713394+0000", 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069733900000000005 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.716082+0000", 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.45422858199999999, 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00079327, 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.241 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716082+0000", 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716082+0000", 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716083+0000", 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2699999999999999e-07 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716090+0000", 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3509999999999996e-06 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716099+0000", 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1439999999999992e-06 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716105+0000", 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3249999999999998e-06 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716137+0000", 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2772999999999999e-05 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.716875+0000", 2024-09-17T07:50:01.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00073804999999999997 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.719775+0000", 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.45053574699999999, 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025856499999999999, 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719775+0000", 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719775+0000", 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719775+0000", 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.72e-07 2024-09-17T07:50:01.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719781+0000", 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5620000000000003e-06 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719788+0000", 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4370000000000001e-06 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719793+0000", 2024-09-17T07:50:01.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5249999999999999e-06 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.719848+0000", 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5463000000000001e-05 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.720034+0000", 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018520599999999999 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.248 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(4.0 e98/97 cost 1048576)", 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.724402+0000", 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.44590885099999999, 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000256469, 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724402+0000", 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724402+0000", 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724402+0000", 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5300000000000004e-07 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724411+0000", 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5900000000000008e-06 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724424+0000", 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2670000000000001e-05 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724429+0000", 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.3519999999999997e-06 2024-09-17T07:50:01.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724462+0000", 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2786000000000001e-05 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.724658+0000", 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019651799999999999 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:01.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:01.253 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:01.254 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:01.253+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:01.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:01.357 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:01.357+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:01.364 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:01.365 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:01.365 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:01.365 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:01.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:01.458 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:01.458+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:01.475 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:01.475 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:01.475 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:01.475 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:01.476 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:01.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:01.559 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:01.559+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:01.587 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:01.587 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:01.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.420392408, 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.588 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:01.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:01.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.41273289199999, 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:01.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:01.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:01.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:01.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.404343324, 2024-09-17T07:50:01.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:01.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:01.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.403972074, 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.604 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:01.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:01.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.395909109, 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.609 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:01.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.39553799700001, 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:01.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:01.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.38625759200001, 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.38459815900001, 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.378573334, 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.37832300700001, 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:01.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.371209169, 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:01.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:01.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.37069207, 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:01.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:01.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.652 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.351720903, 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:01.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:01.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:01.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:01.657 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.34546404300001, 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:01.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.338927981, 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.662 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.966061571, 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.668 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:01.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:01.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:01.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:01.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.964101164, 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.674 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:01.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:01.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:01.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:01.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.07843252799999, 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:01.686 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.76758689, 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:01.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:01.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:01.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:01.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 108.766225445, 2024-09-17T07:50:01.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:01.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:01.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:01.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:01.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:01.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:01.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:01.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:01.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:01.704 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:01.705 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:50:01.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:01.708 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:01.705+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:01.807 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:01.806+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:01.851 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:01.851 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:01.851 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:01.851 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:01.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:01.907 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:01.907+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:02.005 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:02.005 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:02.005 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:02.005 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:02.005 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:02.009 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:02.008+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:02.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:02.109 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:02.109+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:02.157 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:02.157 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:02.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:02.157 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:02.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.897960271, 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:02.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:02.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:02.162 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.529654655, 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:02.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.8374724, 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:02.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:02.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.831615544, 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.176 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.181 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.819922585, 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:02.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.751068002, 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:02.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:02.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:02.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:02.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.739168713, 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.193 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:02.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:02.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:02.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.198 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.688324166, 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:02.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:02.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:02.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.68231595899999, 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.209 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.676615602, 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:02.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.67087597699999, 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:02.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:02.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.337764929, 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.221 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:02.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.226 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.49266403199999, 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:02.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.486577309, 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:02.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.403121008, 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:02.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.396806576, 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:02.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.39043511600001, 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:02.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:02.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.38451264, 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:02.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:02.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 106.326529744, 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:02.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:02.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 102.36882507200001, 2024-09-17T07:50:02.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:02.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:02.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:02.269 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:02.270 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:02.271 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:02.270+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:02.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:02.370+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:02.380 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:02.380 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:02.380 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:02.380 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:02.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:02.474 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:02.474+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:02.514 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:02.528 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:02.528 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:02.528 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:02.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:02.577 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:02.577+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:02.635 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:02.650 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:02.651 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:02.651 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:02.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:02.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:02.673 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:50:02.673 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:02.673+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:02.676 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:02.676 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:02.685 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:02.685 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:02.686 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:02.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:02.774 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:02.774+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:02.784 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:02.784 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:02.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:02.875 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:02.875+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:02.888 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:02.888 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:02.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:02.967 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:02.967 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:02.967 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4] 2024-09-17T07:50:02.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-17T07:50:02.977 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:02.976+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:02.993 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:02.993 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:03.078 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:03.077+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:03.115 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.115 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:03.178 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:03.178+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:03.227 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.227 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:03.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-17T07:50:03.253+0000 7ffb533f1640 -1 mon.a@0(leader).osd e99 definitely_dead 0 2024-09-17T07:50:03.279 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:03.279+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:03.337 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.337 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:03.381 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:03.381+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:03.482 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:03.481+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:03.487 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.487 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:03.583 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:03.583+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:03.638 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.638 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:03.681 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.1. 2024-09-17T07:50:03.684 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:03.683+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:03.753 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.753 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.761 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:03.784 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:03.784+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:03.871 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.872 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:03.885 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:03.885+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:03.979 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:03.979 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:03.986 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:03.986+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:03.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:04.087 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:04.087+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:04.125 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:04.125 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:04.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:04.188 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:04.188+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:04.230 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:04.231 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:04.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:04.322 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:04.322+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:04.376 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:04.376 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:04.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:04.389 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:04.389+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:04.490 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:04.490+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:04.521 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:04.522 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:04.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:04.591 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:04.591+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:04.644 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:04.645 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:04.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:04.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:50:04.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:04.692 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:04.692+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:04.793 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:04.792+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:04.805 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:04.808 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:04.808 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:04.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:04.822 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:04.823 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:04.839 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:04.840 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:04.840 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:04.855 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:04.856 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:04.871 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:04.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:04.875 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:04.894 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:04.894+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.76049129, 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:04.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.752831774, 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.932 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:04.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:04.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.744442206, 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:04.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.744070956, 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.944 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.73600799099999, 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.735636879, 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.726356474, 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.724697041, 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.718672216, 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.718421889, 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.711308051, 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:04.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:04.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:04.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.983 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.710790952, 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:04.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:04.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:04.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.69181978500001, 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:04.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.685562925, 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:04.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:04.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.679026863, 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:04.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:04.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:05.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:05.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:05.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.306160453, 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:05.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:05.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:05.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.30420004600001, 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:05.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:05.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.41853141, 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:05.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:05.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 112.107685772, 2024-09-17T07:50:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:05.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:05.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 112.106324327, 2024-09-17T07:50:05.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:05.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:05.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.035 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:05.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:05.037 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:05.036+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:05.138 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:05.137+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:05.139 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:05.139 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:05.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:05.238 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:05.238+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:05.249 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:05.249 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:05.249 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:05.249 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:05.249 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:05.257 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:05.339 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:05.338+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:05.361 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:05.361 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:05.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:05.361 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:05.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.361 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.10240767400001, 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:05.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.734102058, 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:05.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.041919803, 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.036062947, 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.024369988, 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.955515405, 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.943616116, 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.892771569, 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.886763362, 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.407 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:05.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.881063005, 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.413 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:05.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 116.87532338, 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:05.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 112.54221233200001, 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:05.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:05.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.430 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.697111435, 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:05.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.436 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.691024712, 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:05.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.607568411, 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:05.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:05.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:05.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.60125397900001, 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.594882519, 2024-09-17T07:50:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.588960043, 2024-09-17T07:50:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:05.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:05.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:05.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.530977147, 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:05.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:05.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:05.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 105.573272475, 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:05.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:05.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:05.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:05.478 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:05.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:05.480 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:05.479+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:05.580 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:05.580+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:05.582 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:05.583 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:05.583 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:05.583 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:05.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:05.681 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:05.681+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:05.683 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:50:05.733 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:05.733 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:05.733 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:05.733 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:05.733 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:05.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:05.783 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:05.783+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:05.883 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:05.882+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.884 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.71660680700001, 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:05.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.709267473, 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:05.889 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:05.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:05.893 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.708550399, 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.894 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.699322838, 2024-09-17T07:50:05.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:05.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.69009826999999, 2024-09-17T07:50:05.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:05.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.67462363, 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:05.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:05.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.66533412, 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:05.914 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:05.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.658743875, 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.919 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:05.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.642561394, 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:05.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:05.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.63348647399999, 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:05.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.293047598, 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.932 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:05.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.290995321, 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:05.938 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:05.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:05.943 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.23076367, 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:05.944 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:05.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:05.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:05.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.069584521, 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.950 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:05.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.06956022, 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:05.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.069525844, 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:05.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:05.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.069504487, 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:05.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:05.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.06683584300001, 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.971 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:05.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:05.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:05.976 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.05711827, 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:05.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 106.091007881, 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:05.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:05.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:05.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:05.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:05.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:05.988 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:05.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:05.990 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:05.989+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:06.088 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:06.088 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:06.090 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:06.090+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:06.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:06.192 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:06.191+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:06.233 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:06.233 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:06.233 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:06.234 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:06.234 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:06.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:06.292 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:06.292+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:06.380 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.7112236479999998, 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:06.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.385 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.7082107749999995, 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:06.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.7059438499999997, 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:06.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:06.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.7037742319999998, 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.700484865, 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:06.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.403 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6975916069999997, 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:06.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.407 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6949787360000004, 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:06.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6921404510000002, 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:06.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.416 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6879966120000001, 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:06.420 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6737617560000002, 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.66330730000000004, 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.424 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.65881120299999996, 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.65533403199999996, 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:06.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:06.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.63559792999999998, 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:06.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.62105750800000004, 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:06.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:06.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.61732437299999998, 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.61356257999999997, 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.446 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:06.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:06.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.60628269000000001, 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.59190790400000004, 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:06.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.457 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.57795202999999995, 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:06.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:06.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:06.461 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:06.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:06.463 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:06.463+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:06.564 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:06.563+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:06.577 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:06.577 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:06.577 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:06.577 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:06.586 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:06.672 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:06.670+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:06.730 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:06.730 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:06.730 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:06.730 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:06.731 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:06.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:06.780 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:06.779+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:06.867 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:06.867+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:06.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:06.882 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:06.882 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:06.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:06.882 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:06.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.71560701, 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:06.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.707947494, 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.888 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:06.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:06.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:06.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.699557926, 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:06.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:06.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:06.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.699186676, 2024-09-17T07:50:06.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:06.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.691123711, 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.906 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:06.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:06.910 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.69075259900001, 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:06.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:06.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.68147219399999, 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:06.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:06.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:06.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:06.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.67981276099999, 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:06.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.673787936, 2024-09-17T07:50:06.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:06.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:06.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.67353760899999, 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:06.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.666423771, 2024-09-17T07:50:06.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:06.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:06.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.66590667200001, 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:06.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.947 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.646935505, 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.64067864499999, 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:06.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.956 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.634142583, 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:06.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:06.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:06.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.261276173, 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:06.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:06.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:06.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:06.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.259315766, 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:06.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:06.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.37364712999999, 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:06.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.980 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:06.981 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:06.981 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.06280149200001, 2024-09-17T07:50:06.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:06.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.981 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:06.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:06.988 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.061440047, 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:06.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:06.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:06.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:06.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:06.996 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:06.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:06.998 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:06.998 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:06.999 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:06.999 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:06.996+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:07.008 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:07.009 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:07.009 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:07.024 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:07.025 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:07.041 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:07.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:07.045 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:07.098 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:07.098+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:07.111 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.111 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:07.199 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:07.199+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:07.221 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.221 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:07.301 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:07.300+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:07.332 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.332 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:07.402 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:07.402+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:07.450 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.451 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:07.502 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:07.502+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:07.557 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.558 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:07.604 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:07.604+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:07.663 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.663 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:07.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:50:07.705 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:07.704+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:07.770 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.770 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.778 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:07.805 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:07.805+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:07.876 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.876 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:07.928 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:07.928+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:07.982 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:07.982 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:07.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:08.034 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:08.034+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:08.092 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.093 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:08.109 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:08.109+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:08.210 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:08.210+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:08.243 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.243 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:08.313 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:08.313+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:08.393 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.393 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:08.446 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:08.446+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:08.503 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.504 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.512 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:08.517 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:08.517+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:08.618 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:08.618+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:08.653 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.653 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:08.698 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4] out_osds: [5, 1] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:08.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:08.698 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T07:50:08.698 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-09-17T07:50:08.699 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0 2024-09-17T07:50:08.702 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:50:08.721 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:08.720+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:08.773 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.773 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:08.821 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:08.821+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:08.892 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.892 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:08.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:08.925 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:08.925+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:08.999 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:08.999 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:09.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:09.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:09.051 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:09.051+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:09.106 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:09.106 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:09.115 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:09.148 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:09.156 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:09.156+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:09.163 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:09.179 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:09.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:09.179 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:09.196 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:09.212 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:09.212 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:09.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:09.229 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:09.229+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:09.255 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:09.255 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:09.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:09.330 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:09.330+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:09.360 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:09.360 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:09.360 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:09.360 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:09.361 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:09.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:09.432 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:09.431+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:09.467 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7984028349999992, 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:09.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7953899619999998, 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:09.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:09.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:09.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7931230370000009, 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:09.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:09.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:09.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7909534190000009, 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:09.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.484 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7876640520000002, 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:09.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:09.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7847707939999999, 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:09.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7821579229999998, 2024-09-17T07:50:09.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:09.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:09.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7793196380000005, 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:09.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.502 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7751757989999994, 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:09.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:09.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.7609409429999996, 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:09.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:09.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7504864869999999, 2024-09-17T07:50:09.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:09.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7459903900000002, 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:09.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7425132190000001, 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:09.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.522 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7227771170000001, 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:09.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7082366950000001, 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:09.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:09.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.70450356, 2024-09-17T07:50:09.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.7007417670000002, 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:09.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.6934618769999998, 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:09.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.543 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.679087091, 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:09.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.6651312169999999, 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:09.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:09.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:09.552 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:09.553 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:09.553+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:09.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:09.653 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:09.653+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:09.658 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:09.658 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:09.658 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:09.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.3 primary-affinity to 0 (802) 2024-09-17T07:50:09.755 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:09.754+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:09.804 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:09.804 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:09.804 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:09.804 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:09.804 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:09.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:09.855 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:09.855+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:09.915 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.749109899, 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.741450383, 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:09.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:09.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.733060815, 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:09.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.732689565, 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:09.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:09.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.72462659999999, 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:09.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.724255488, 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.714975083, 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:09.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:09.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.71331565, 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:09.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.707290825, 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:09.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.707040498, 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:09.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:09.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:09.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:09.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.69992666, 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:09.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:09.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:09.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:09.981 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.699409561, 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:09.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:09.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.68043839400001, 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.988 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:09.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:09.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:09.993 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.674181534, 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:09.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:09.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:09.999 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.667645472, 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:10.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:10.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.006 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.294779062, 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:10.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:10.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.292818655, 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:10.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:10.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.407150019, 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.021 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:10.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:10.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.028 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.096304381, 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:10.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:10.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.035 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.094942936, 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:10.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.042 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:10.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:10.044 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:10.043+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:10.144 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:10.143+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5371.0:147 3.0 3:11e6e56e:::rbd_object_map.123eb9c2b614.0000000000000006:head [writefull 0~38 in=38b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e102)", 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:06.514026+0000", 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.632351844, 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.63237627, 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5371", 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2813598560", 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 147 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514026+0000", 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514026+0000", 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514028+0000", 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.516e-06 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514033+0000", 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.648e-06 2024-09-17T07:50:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514043+0000", 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0105000000000001e-05 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514049+0000", 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5149999999999997e-06 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514114+0000", 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5140000000000003e-05 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-09-17T07:50:10.150 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:10.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:10.245 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:10.245+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:10.296 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:10.296 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:10.296 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:10.296 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:10.296 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:10.304 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:10.346 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:10.345+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.18895388799999, 2024-09-17T07:50:10.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:10.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:10.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.820648272, 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:10.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:10.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.12846601699999, 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.457 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.122609161, 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.110916202, 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.04206161899999, 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.03016233, 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.979317783, 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:10.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:10.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.97330957600001, 2024-09-17T07:50:10.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:10.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:10.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.967609219, 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:10.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:10.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:10.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.96186959400001, 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.504 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:10.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:10.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:10.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.628758546, 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:10.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:10.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:10.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:10.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.78365764900001, 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.516 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:10.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:10.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.777570926, 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.522 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:10.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.694114625, 2024-09-17T07:50:10.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:10.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.687800193, 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:10.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.681428733, 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:10.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:10.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.545 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.675506257, 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:10.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:10.551 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 114.617523361, 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:10.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:10.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:10.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.65981868900001, 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:10.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:10.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:10.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:10.561 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:10.564 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:10.447+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:10.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:10.566 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:10.566+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:10.667 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:10.667+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:10.675 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:10.675 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:10.675 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:10.675 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:10.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:10.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:50:10.768 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:10.768+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:10.782 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:10.782 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:10.782 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:10.782 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:10.782 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:10.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:10.869 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:10.868+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:10.890 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:10.890 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:10.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:10.890 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:10.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.890 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.72371297199999, 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:10.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.71637363799999, 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:10.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:10.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:10.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:10.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:10.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.902 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.902 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.902 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.715656564, 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:10.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:10.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:10.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:10.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:10.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:10.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.70642900300001, 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:10.916 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.697204435, 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:10.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:10.920 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.681729795, 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:10.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:10.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:10.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:10.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:10.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:10.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:10.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:10.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:10.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:10.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:10.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:10.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.929 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.929 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.930 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:10.930 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.67244028499999, 2024-09-17T07:50:10.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:10.930 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.930 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.930 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.931 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:10.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.931 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:10.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:10.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:10.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:10.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:10.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:10.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:10.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:10.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.66585004000001, 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:10.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:10.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.942 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.64966755899999, 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.943 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:10.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.640592639, 2024-09-17T07:50:10.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:10.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.300153763, 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:10.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:10.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:10.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.29810148599999, 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:10.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:10.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:10.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:10.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:10.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.237869835, 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:10.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:10.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:10.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:10.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:10.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.07669068600001, 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:10.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.076666385, 2024-09-17T07:50:10.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:10.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:10.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.07663200899999, 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.986 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:10.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:10.991 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.076610652, 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:10.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:10.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:10.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.997 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.073942008, 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:10.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:10.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:11.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.064224435, 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:11.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:11.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:11.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.09811404600001, 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:11.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.014 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.015 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:11.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:11.017 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:11.015+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:11.115 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:11.115 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:11.115 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:11.115 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:11.116 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:11.116+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:11.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:11.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:11.218 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:11.217+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:11.220 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:11.220 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:11.220 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:11.221 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:11.221 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:11.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:11.318 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:11.318+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:11.320 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:11.327 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.327 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.658015778999999, 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:11.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.655002906, 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:11.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.652735980999999, 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:11.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.650566362999999, 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.340 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.647276996, 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.344 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:11.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:11.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.644383738, 2024-09-17T07:50:11.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:11.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.641770867, 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:11.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:11.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.638932582000001, 2024-09-17T07:50:11.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:11.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:11.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:11.360 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.634788743, 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:11.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:11.364 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.620553887, 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:11.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6100994310000001, 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.371 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6056033339999995, 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:11.375 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6021261630000003, 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:11.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:11.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5823900609999999, 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:11.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:11.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:11.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5678496390000003, 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:11.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5641165040000002, 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:11.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:11.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5603547110000004, 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:11.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.553074821, 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:11.394 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:11.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:11.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5387000349999997, 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:11.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.401 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.5247441610000001, 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:11.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:11.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:11.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:11.406 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:11.407 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:11.408 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:11.408 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:11.408 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:11.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:11.409 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:11.409 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:11.410 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:11.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:11.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:11.453 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:11.453+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:11.499 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:50:11.520 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:11.520+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:11.560 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:11.560 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:11.569 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:11.653 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:11.653+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:11.705 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:50:11.704+0000 7f0578e4a640 0 -- 172.21.15.98:0/2998672302 >> [v2:172.21.15.26:6811/31886,v1:172.21.15.26:6813/31886] conn(0x7f0554008980 msgr2=0x7f055c045e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:50:11.712 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:11.712 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:11.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:11.762 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:11.762+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:11.823 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:11.823+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:11.864 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:11.865 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:11.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:11.924 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:11.924+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:12.014 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.014 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:12.025 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:12.025+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.123 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.125 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:12.125+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:12.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:12.226 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:12.226+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.273 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:12.328 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:12.328+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:12.391 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.391 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:12.445 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:12.445+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:12.499 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.499 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:12.530 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:12.529+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:12.631 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:12.630+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:12.645 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.645 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:12.732 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:12.731+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:12.750 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.750 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:12.833 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:12.832+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:12.856 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.856 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:12.934 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:12.933+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:12.962 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:12.962 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:12.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:13.035 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:13.034+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:13.081 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:13.081 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:13.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:13.135 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:13.135+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:13.233 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:13.237 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:13.236+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:13.241 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:13.338 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:13.337+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:13.342 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:13.342 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:13.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:13.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:13.439 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:13.439+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:13.493 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:13.494 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:13.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:13.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:50:13.536 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:13.539 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:13.539+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:13.554 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:13.574 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:13.634 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:13.634 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:13.634 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:13.635 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:13.636 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:13.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:13.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:13.677 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:13.677+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:13.740 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:13.740+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:13.782 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:13.782 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:13.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:13.782 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 131.523614501, 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:13.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:13.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.155308885, 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.787 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:13.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.46312663, 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:13.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:13.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:13.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.457269774, 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.798 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:13.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:13.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.445576815, 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:13.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:13.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.37672223200001, 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.810 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:13.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:13.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.36482294299999, 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:13.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.313978396, 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.307970189, 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.30226983199999, 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.296530207, 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.963419159, 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.118318262, 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.11223153900001, 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.02877523799999, 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.022460806, 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.016089346, 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 118.01016687000001, 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 117.95218397399999, 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 113.994479302, 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:13.899 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:13.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:13.901 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:13.900+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:14.001 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:14.000+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:14.010 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:14.010 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:14.010 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:14.010 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:14.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:14.102 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:14.102+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:14.116 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:14.116 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:14.116 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:14.116 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:14.116 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:14.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:14.202 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:14.202+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:14.223 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:14.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.056884462, 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:14.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:14.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.04954512800001, 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:14.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:14.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:14.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.04882805400001, 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:14.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:14.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.03960049299999, 2024-09-17T07:50:14.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:14.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.03037592499999, 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:14.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:14.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.01490128500001, 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:14.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:14.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:14.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.00561177500001, 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.251 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.99902152999999, 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.98283904900001, 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:14.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.263 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.97376412899999, 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:14.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:14.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:14.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:14.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.633325253, 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:14.268 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:14.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:14.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:14.274 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.63127297600001, 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:14.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:14.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:14.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:14.280 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.571041325, 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:14.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:14.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:14.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 121.409862176, 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:14.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 121.40983787499999, 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:14.292 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 121.40980349900001, 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.297 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:14.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 121.409782142, 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:14.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 121.407113498, 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:14.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.312 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 121.397395925, 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:14.313 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:14.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.431285536, 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:14.318 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:14.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:14.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:14.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.324 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:14.326 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:14.326 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:14.325+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:14.426 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:14.426+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:14.464 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:14.464 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:14.464 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:14.464 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:14.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:14.556 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:14.556+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:14.610 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:14.610 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:14.610 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:14.610 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:14.610 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:14.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:14.659 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:14.659+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:14.716 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4] out_osds: [5, 1] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:14.716 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:14.717 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-17T07:50:14.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-17T07:50:14.718 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:50:14.729 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:14.729+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:14.758 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.089074973000001, 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.086062099999999, 2024-09-17T07:50:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:14.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.083795175000001, 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:14.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.081625557000001, 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.772 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.776 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.07833619, 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:14.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:14.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:14.781 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.075442932, 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:14.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:14.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.072830060999999, 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:14.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:14.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.069991776, 2024-09-17T07:50:14.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:14.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.794 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.065847936999999, 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.051613080999999, 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:14.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0411586249999996, 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:14.802 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:14.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0366625280000008, 2024-09-17T07:50:14.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:14.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0331853570000007, 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:14.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.0134492549999994, 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:14.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9989088329999998, 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9951756980000006, 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.821 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.824 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9914139049999999, 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.828 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9841340150000004, 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9697592289999992, 2024-09-17T07:50:14.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.9558033550000005, 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:14.839 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:14.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:14.842 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:14.840+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:14.942 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:14.941+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:14.944 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:14.944 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:14.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:15.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-17T07:50:15.011+0000 7ffb533f1640 -1 mon.a@0(leader).osd e106 definitely_dead 0 2024-09-17T07:50:15.042 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:15.042+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:15.054 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:15.055 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:15.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:15.143 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:15.143+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.99903765299999, 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.991378137, 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.98298856900001, 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.98261731900001, 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.97455435399999, 2024-09-17T07:50:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.97418324200001, 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.96490283700001, 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.963243404, 2024-09-17T07:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.957218579, 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.956968252, 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.94985441399999, 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.94933731500001, 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.93036614799999, 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.92410928800001, 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.917573226, 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.544706816, 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.542746409, 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.657077773, 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.34623213499999, 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.34487068999999, 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.275 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:15.276 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:15.277 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:15.276+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:15.377 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:15.377+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5371.0:147 3.0 3:11e6e56e:::rbd_object_map.123eb9c2b614.0000000000000006:head [writefull 0~38 in=38b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e102)", 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:06.514026+0000", 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.9052125560000004, 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9052283800000005, 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-09-17T07:50:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5371", 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2813598560", 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 147 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514026+0000", 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514026+0000", 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514028+0000", 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.516e-06 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514033+0000", 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.648e-06 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514043+0000", 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0105000000000001e-05 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514049+0000", 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5149999999999997e-06 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:06.514114+0000", 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5140000000000003e-05 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:50:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-09-17T07:50:15.423 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:15.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:15.512 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:15.512+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:15.569 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:15.569 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:15.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:15.619 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:15.618+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:15.638 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:15.681 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:15.681+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.1. 2024-09-17T07:50:15.728 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.468788717, 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.10048310100001, 2024-09-17T07:50:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.408300846, 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.40244398999999, 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.39075103099999, 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.321896448, 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.30999715900001, 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.25915261199999, 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.253144405, 2024-09-17T07:50:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.24744404800001, 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 127.241704423, 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.908593375, 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.063492478, 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.057405755, 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.97394945400001, 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.967635022, 2024-09-17T07:50:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.961263562, 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.955341086, 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.89735819000001, 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 115.939653518, 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:15.861 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:15.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:15.864 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-17T07:50:15.864 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:15.864 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:15.865 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:15.865 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:15.865 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:15.866 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:15.866 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:15.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:15.866 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:15.867 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:15.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:15.868 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:15.862+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:15.963 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:15.963+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:15.976 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:15.976 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:15.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:16.064 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:16.064+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:16.083 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.084 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:16.166 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.165+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:16.190 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.190 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:16.267 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:16.266+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:16.337 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.338 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:16.389 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.389+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:16.444 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.444 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.452 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:16.493 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:16.493+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:16.570 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.569+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:16.590 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.590 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:16.671 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.671+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:16.708 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.708 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:16.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:50:16.772 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.772+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:16.817 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.817 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:16.873 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.872+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:16.961 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:16.961 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:16.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:16.973 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:16.972+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:17.071 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.071 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.073 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:17.073+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:17.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:17.175 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:17.174+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:17.221 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.221 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:17.276 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:17.276+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:17.372 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.372 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.377 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:17.377+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:17.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:17.478 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:17.478+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:17.485 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.485 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:17.501 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:50:17.579 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:17.579+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:17.633 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.633 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:17.683 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:17.683+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:17.781 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.781 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.781 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:17.781+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:17.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:17.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:17.912 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:17.912+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:17.926 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:17.926 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:17.934 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:17.971 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:17.982 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:17.982+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:17.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:18.003 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:18.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:18.003 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:18.018 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:18.019 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:18.032 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:18.032 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:18.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:18.035 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:18.039 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:18.041 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:18.085 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.085+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.471447633, 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.141 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:18.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.468434760000001, 2024-09-17T07:50:18.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:18.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:18.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.466167835, 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:18.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.463998217, 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.46070885, 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.457815591999999, 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.455202720999999, 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.452364436, 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.448220596999999, 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.433985741000001, 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.423531284999999, 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.419035188000001, 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.415558017, 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.395821915000001, 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.381281492999999, 2024-09-17T07:50:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.377548358, 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.373786565, 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.366506675, 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.352131889000001, 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.338176015, 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:18.218 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:18.220 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:18.219+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:18.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:18.321 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.320+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.323 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:18.323 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:18.323 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:18.324 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:18.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:18.422 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.421+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.433 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:18.433 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:18.433 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:18.433 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:18.433 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:18.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:18.523 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.523+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.378039935, 2024-09-17T07:50:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.37038041900001, 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:18.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:18.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:18.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.361990851, 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.361619601, 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.35355663600001, 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.353185524, 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.343905119, 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.34224568600001, 2024-09-17T07:50:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.33622086099999, 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.33597053400001, 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.328856696, 2024-09-17T07:50:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.328339597, 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.30936843000001, 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.30311157, 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.29657550799999, 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.92370909799999, 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.921748691, 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.03608005500001, 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.725234417, 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 125.723872972, 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.651 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:18.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:18.653 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.652+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.753 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.753+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.755 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:18.755 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:18.755 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:18.755 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:18.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:18.854 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:18.854+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:18.864 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:18.865 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:18.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:18.955 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:18.955+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:18.976 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.717517684, 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 131.34921206800001, 2024-09-17T07:50:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.65702981300001, 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.651172957, 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.63947999800001, 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.57062541499999, 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.55872612600001, 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.50788157900001, 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.50187337200001, 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.49617301500001, 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 130.49043338999999, 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.157322342, 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.31222144500001, 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:19.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.306134722, 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:19.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.222678421, 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:19.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:19.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.064 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.216363989, 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:19.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:19.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:19.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.209992529, 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:19.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:19.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:19.076 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.204070053, 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:19.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:19.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.146087157, 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:19.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:19.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:19.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.088 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 119.18838248500001, 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:19.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:19.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:19.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:19.093 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:19.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:19.095 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:19.094+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:19.195 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:19.195+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:19.203 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:19.203 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:19.203 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:19.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:19.297 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:19.297+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:19.310 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:19.310 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:19.310 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:19.310 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:19.310 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:19.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:19.397 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:19.397+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:19.418 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:19.418 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:19.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:19.418 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:19.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.418 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.251795399, 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:19.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.24445606500001, 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.423 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:19.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.427 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.24373899099999, 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:19.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:19.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.432 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.23451143, 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:19.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:19.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.22528686199999, 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.20981222200001, 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.20052271200001, 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.193932467, 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.17774998600001, 2024-09-17T07:50:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.16867506599999, 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.82823619000001, 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.468 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.82618391299999, 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:19.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:19.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.76595226200001, 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:19.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:19.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.480 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.60477311299999, 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:19.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.604748812, 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.486 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:19.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:19.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.60471443599999, 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:19.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:19.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:19.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:19.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.604693079, 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:19.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.602024435, 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:19.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:19.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.592306862, 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:19.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 119.62619647299999, 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.511 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:19.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:19.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.516 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.517 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.517 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:19.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:19.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:50:19.519 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:19.518+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:19.619 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:19.618+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:19.658 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:19.658 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:19.658 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:19.658 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:19.666 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:19.720 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:19.719+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:19.763 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:19.764 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:19.764 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:19.764 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:19.764 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:19.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:19.820 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:19.820+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:19.870 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:19.870 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:19.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:19.870 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:19.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.870 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.201042375, 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:19.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.874 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.198029502000001, 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:19.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:19.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.195762577, 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.193592959, 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.883 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:19.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.190303591999999, 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:19.887 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:19.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:19.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:19.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.187410333999999, 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:19.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.895 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.184797462999999, 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.181959178, 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:19.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.177815338999999, 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.904 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 19.163580483, 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.153126027000001, 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:19.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:19.915 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.14862993, 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:19.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:19.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.145152759, 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:19.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.922 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.125416657000001, 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:19.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.110876234999999, 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.1071431, 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.103381306999999, 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:19.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:19.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.096101417, 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:19.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.081726631, 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:19.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.067770757, 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:19.949 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:19.949 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:19.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:19.951 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:19.950+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:20.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:20.053 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:20.053 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:20.053 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:20.053 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:20.053 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:20.053+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:20.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:20.144 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:20.151 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:20.150+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:20.160 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:20.160 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:20.176 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:20.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:20.177 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:20.193 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:20.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:20.206 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:20.206 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:20.208 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:20.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:20.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:20.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:20.252 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:20.251+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:20.353 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:20.352+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:20.356 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:20.357 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:20.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:20.454 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:20.454+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:20.507 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:20.507 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:20.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:20.556 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:20.556+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:20.656 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:20.655+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:20.657 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:20.657 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:20.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:20.729 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:50:20.757 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:20.757+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:20.810 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:20.810 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:20.819 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:20.864 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:20.864+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:20.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:20.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:20.865 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-17T07:50:20.865 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:50:20.865 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:50:20.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-17T07:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:20.938 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:20.966 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:20.966 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-09-17T07:50:20.966 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:20.979 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:20.979+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:21.059 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:21.059+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:21.160 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:21.160+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:21.275 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:21.275+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.337 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:21.378 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:21.378+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:21.463 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:21.462+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:21.564 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:21.563+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:21.672 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:21.671+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:21.737 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.737 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:21.765 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:21.765+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:21.866 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:21.866+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:21.887 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.888 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:21.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:21.966 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:21.965+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:21.998 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:21.998 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:22.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:22.090 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:22.089+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:22.148 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:22.148 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:22.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:22.198 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:22.197+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:22.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:22.269 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:22.269+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:22.300 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:22.300 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:22.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:22.343 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:22.359 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:22.359 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:22.369 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:22.369+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:22.375 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:22.375 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:22.375 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:22.392 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:22.392 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:22.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:22.408 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:22.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:22.415 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:22.415 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:22.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:22.415 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.15650149699999, 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:22.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.78819588100001, 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:22.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:22.425 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.096013626, 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:22.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:22.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:22.431 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.09015676999999, 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:22.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.07846381100001, 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:22.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:22.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.446 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.00960922799999, 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:22.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:22.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.452 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.997709939, 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:22.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.94686539200001, 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:22.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:22.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:22.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.940857185, 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.464 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:22.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:22.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.935156828, 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.471 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:22.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:22.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.476 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.92941720300001, 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:22.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 129.59630615500001, 2024-09-17T07:50:22.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:22.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:22.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.751205258, 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:22.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:22.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.745118535, 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.494 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:22.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:22.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.661662234, 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.500 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:22.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.65534780199999, 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.506 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:22.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:22.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.648976342, 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:22.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:22.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.517 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.643053866, 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:22.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:22.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.523 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 126.58507097, 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:22.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:22.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.627366298, 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:22.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:22.533 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:22.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:22.535 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:22.534+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:22.636 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:22.635+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:22.645 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:22.645 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:22.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:22.737 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:22.735+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:22.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:50:22.791 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:22.792 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:22.792 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:22.792 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:22.792 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:22.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:22.844 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:22.844+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.903 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.73641896199999, 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:22.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.72907962799999, 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:22.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:22.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.728362554, 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:22.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.71913499300001, 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:22.917 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.709910425, 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:22.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:22.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:22.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.694435785, 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.926 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.68514627499999, 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.930 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:22.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.67855603000001, 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:22.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.938 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.66237354899999, 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:22.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:22.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.65329862900001, 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:22.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:22.947 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.312859753, 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:22.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:22.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.31080747600001, 2024-09-17T07:50:22.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:22.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:22.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.250575825, 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:22.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:22.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:22.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.964 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.08939667600001, 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:22.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.08937237500001, 2024-09-17T07:50:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.08933799900001, 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.089316642, 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.08664799799999, 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.076930425, 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.11082003600001, 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.000 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:23.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:23.002 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:22.999+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:23.102 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:23.102+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:23.140 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:23.140 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:23.141 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:23.141 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:23.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:23.234 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:23.234+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:23.286 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:23.286 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:23.286 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:23.286 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:23.286 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:23.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:23.304 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:23.303+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:23.404 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:23.404+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.763694068, 2024-09-17T07:50:23.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.760681195, 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:23.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.758414269999999, 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.444 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.756244651999999, 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:23.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:23.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.752955284999999, 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.449 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:23.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:23.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.750062026999998, 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.747449155999998, 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.461 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.744610870999999, 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:23.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.740467032000002, 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:23.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:23.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.726232176, 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:23.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.715777719999998, 2024-09-17T07:50:23.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:23.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.711281623000001, 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.479 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.707804452000001, 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:23.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:23.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.688068350000002, 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:23.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.673527927999999, 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:23.490 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.669794793000001, 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.666032999999999, 2024-09-17T07:50:23.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.497 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.658753109999999, 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:23.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.644378324000002, 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:23.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.630422450000001, 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:23.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:23.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:23.509 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:23.510 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:23.510+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:23.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:23.610 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:23.610+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:23.618 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:23.618 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:23.618 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:23.619 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:23.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:23.711 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:23.710+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:23.728 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:23.729 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:23.729 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:23.729 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:23.729 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:23.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:23.739 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:50:23.835 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:23.834+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:23.893 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.726920329, 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:23.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.71926081300001, 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:23.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:23.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.71087124499999, 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:23.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.71049999499999, 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:23.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.913 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.70243703, 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.914 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:23.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:23.918 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.70206591799999, 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.69278551299999, 2024-09-17T07:50:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:23.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:23.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:23.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.69112608, 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:23.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:23.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:23.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.68510125500001, 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:23.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.684850928, 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:23.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.67773708999999, 2024-09-17T07:50:23.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.67721999099999, 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.658248824, 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.65199196399999, 2024-09-17T07:50:23.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:23.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:23.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.64545590200001, 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.27258949200001, 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:23.967 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:23.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:23.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:23.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.270629085, 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:23.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:23.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:23.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.384960449, 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 131.07411481099999, 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:23.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:23.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 131.072753366, 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:23.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:23.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:23.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:23.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:23.997 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:23.998 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:23.998+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:23.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:24.099 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:24.099+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:24.142 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:24.142 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:24.142 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:24.142 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:24.150 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:24.200 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:24.199+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:24.252 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:24.252 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:24.252 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:24.252 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:24.252 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:24.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:24.301 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:24.300+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:24.402 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:24.400+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:24.403 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 142.145009641, 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.408 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.77670402499999, 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:24.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.08452177000001, 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.413 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:24.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.078664914, 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.06697195500001, 2024-09-17T07:50:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:24.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.998117372, 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.431 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.98621808300001, 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:24.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:24.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:24.443 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.93537353599999, 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:24.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.449 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.92936532900001, 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.92366497200001, 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:24.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:24.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.91792534699999, 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:24.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 131.58481429899999, 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.73971340200001, 2024-09-17T07:50:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:24.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:24.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.733626679, 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:24.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:24.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.65017037800001, 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.489 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:24.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:24.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.643855946, 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:24.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.63748448600001, 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.63156201000001, 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.57357911400001, 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.61587444200001, 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:24.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:24.523 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:24.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:24.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:24.526 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:24.523+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:24.625 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:24.625+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:24.633 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:24.634 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:24.634 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:24.634 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:24.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:24.681 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:24.727 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:24.726+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:24.783 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:24.783 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:24.783 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:24.783 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:24.783 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:24.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:24.836 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:24.836+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.72419057900001, 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:24.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:24.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.71685124499999, 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:24.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:24.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.71613417099999, 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:24.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.70690661, 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:24.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.910 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.697682042, 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.911 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:24.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:24.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.68220740199999, 2024-09-17T07:50:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.67291789199999, 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:24.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.666327647, 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.924 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:24.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.65014516599999, 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.929 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:24.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.641070246, 2024-09-17T07:50:24.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:24.937 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.30063136999999, 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:24.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:24.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.944 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.298579093, 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:24.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:24.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:24.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.23834744199999, 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:24.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:24.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:24.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.077168293, 2024-09-17T07:50:24.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:24.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.077143992, 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:24.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:24.967 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.077109616, 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.968 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:24.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:24.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.07708825899999, 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:24.973 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:24.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.07441961500001, 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:24.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:24.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.06470204199999, 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:24.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:24.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.098591653, 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:24.988 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:24.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:24.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:24.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:24.994 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:24.995 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:24.996 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:24.997 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:24.997 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:24.995+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:24.998 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:24.999 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:25.098 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:25.098+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:25.136 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.137 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:25.200 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:25.199+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:25.242 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.242 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:25.300 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:25.299+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:25.348 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.348 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:25.401 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:25.400+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:25.464 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.464 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:25.513 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:25.512+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:25.602 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:25.601+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:25.613 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.613 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:25.706 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:25.705+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:25.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:50:25.764 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.765 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:25.804 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:25.804+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:25.905 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:25.904+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:25.917 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:25.917 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:25.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:25.975 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:25.975 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:25.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:50:26.006 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:26.006+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:26.069 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.069 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.078 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:26.107 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:26.106+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:26.184 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.184 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.193 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:26.237 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:26.237+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:50:26.288 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:50:26.299 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-17T07:50:25.453093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215549411,"num_objects":98,"num_object_clones":1,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":6,"num_read":3892,"num_read_kb":175713,"num_write":683,"num_write_kb":264996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":83886434,"num_keys_recovered":15,"num_objects_omap":12,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":704,"ondisk_log_size":704,"up":26,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1875360,"kb_used_data":355292,"kb_used_omap":8,"kb_used_meta":1519991,"kb_avail":469983840,"statfs":{"total":483183820800,"available":481263452160,"internally_reserved":0,"allocated":363819008,"data_stored":363951415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8520,"internal_metadata":1556471480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.953656"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":32,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.505053+0000","last_change":"2024-09-17T07:50:17.504244+0000","last_active":"2024-09-17T07:50:17.505053+0000","last_peered":"2024-09-17T07:50:17.505053+0000","last_clean":"2024-09-17T07:50:17.505053+0000","last_became_active":"2024-09-17T07:50:17.503881+0000","last_became_peered":"2024-09-17T07:50:17.503881+0000","last_unstale":"2024-09-17T07:50:17.505053+0000","last_undegraded":"2024-09-17T07:50:17.505053+0000","last_fullsized":"2024-09-17T07:50:17.505053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:17:21.810066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"45'7","reported_seq":80,"reported_epoch":48,"state":"stale+active+undersized+degraded","last_fresh":"2024-09-17T07:48:05.372036+0000","last_change":"2024-09-17T07:48:01.879165+0000","last_active":"2024-09-17T07:48:05.372036+0000","last_peered":"2024-09-17T07:48:05.372036+0000","last_clean":"2024-09-17T07:47:41.593387+0000","last_became_active":"2024-09-17T07:47:45.602779+0000","last_became_peered":"2024-09-17T07:47:45.602779+0000","last_unstale":"2024-09-17T07:48:08.649038+0000","last_undegraded":"2024-09-17T07:48:01.878970+0000","last_fullsized":"2024-09-17T07:47:45.602289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:18.255015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":33,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-17T07:50:17.501658+0000","last_change":"2024-09-17T07:50:16.556015+0000","last_active":"2024-09-17T07:50:17.501658+0000","last_peered":"2024-09-17T07:50:17.501658+0000","last_clean":"2024-09-17T07:50:17.501658+0000","last_became_active":"2024-09-17T07:50:16.504245+0000","last_became_peered":"2024-09-17T07:50:16.504245+0000","last_unstale":"2024-09-17T07:50:17.501658+0000","last_undegraded":"2024-09-17T07:50:17.501658+0000","last_fullsized":"2024-09-17T07:50:17.501658+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:34.658470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":756,"num_read_kb":27413,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12583028,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-17T07:50:17.506196+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:50:17.506196+0000","last_peered":"2024-09-17T07:50:17.506196+0000","last_clean":"2024-09-17T07:50:17.506196+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:50:17.506196+0000","last_undegraded":"2024-09-17T07:50:17.506196+0000","last_fullsized":"2024-09-17T07:50:17.506196+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":34,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.500458+0000","last_change":"2024-09-17T07:50:00.721078+0000","last_active":"2024-09-17T07:50:17.500458+0000","last_peered":"2024-09-17T07:50:17.500458+0000","last_clean":"2024-09-17T07:50:17.500458+0000","last_became_active":"2024-09-17T07:50:00.666246+0000","last_became_peered":"2024-09-17T07:50:00.666246+0000","last_unstale":"2024-09-17T07:50:17.500458+0000","last_undegraded":"2024-09-17T07:50:17.500458+0000","last_fullsized":"2024-09-17T07:50:17.500458+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:58.292891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"58'257","reported_seq":1217,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:50:17.500590+0000","last_change":"2024-09-17T07:48:01.916903+0000","last_active":"2024-09-17T07:50:17.500590+0000","last_peered":"2024-09-17T07:50:17.500590+0000","last_clean":"2024-09-17T07:47:43.970152+0000","last_became_active":"2024-09-17T07:47:45.604092+0000","last_became_peered":"2024-09-17T07:47:45.604092+0000","last_unstale":"2024-09-17T07:50:17.500590+0000","last_undegraded":"2024-09-17T07:48:01.915803+0000","last_fullsized":"2024-09-17T07:47:45.603408+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:28.677664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797533,"num_objects":17,"num_object_clones":1,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1339,"num_read_kb":77312,"num_write":306,"num_write_kb":58504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":94,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.504881+0000","last_change":"2024-09-17T07:50:17.504262+0000","last_active":"2024-09-17T07:50:17.504881+0000","last_peered":"2024-09-17T07:50:17.504881+0000","last_clean":"2024-09-17T07:50:17.504881+0000","last_became_active":"2024-09-17T07:50:17.503909+0000","last_became_peered":"2024-09-17T07:50:17.503909+0000","last_unstale":"2024-09-17T07:50:17.504881+0000","last_undegraded":"2024-09-17T07:50:17.504881+0000","last_fullsized":"2024-09-17T07:50:17.504881+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:38:28.806823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"45'5","reported_seq":87,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.500600+0000","last_change":"2024-09-17T07:48:04.736433+0000","last_active":"2024-09-17T07:50:17.500600+0000","last_peered":"2024-09-17T07:50:17.500600+0000","last_clean":"2024-09-17T07:50:17.500600+0000","last_became_active":"2024-09-17T07:48:04.643512+0000","last_became_peered":"2024-09-17T07:48:04.643512+0000","last_unstale":"2024-09-17T07:50:17.500600+0000","last_undegraded":"2024-09-17T07:50:17.500600+0000","last_fullsized":"2024-09-17T07:50:17.500600+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:58.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"109'196","reported_seq":1000,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-17T07:50:21.514389+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:50:21.514389+0000","last_peered":"2024-09-17T07:50:21.514389+0000","last_clean":"2024-09-17T07:50:21.514389+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:50:21.514389+0000","last_undegraded":"2024-09-17T07:50:21.514389+0000","last_fullsized":"2024-09-17T07:50:21.514389+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:00.953341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777417,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1412,"num_read_kb":46188,"num_write":101,"num_write_kb":32778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.504497+0000","last_change":"2024-09-17T07:50:17.503770+0000","last_active":"2024-09-17T07:50:17.504497+0000","last_peered":"2024-09-17T07:50:17.504497+0000","last_clean":"2024-09-17T07:50:17.504497+0000","last_became_active":"2024-09-17T07:50:17.503382+0000","last_became_peered":"2024-09-17T07:50:17.503382+0000","last_unstale":"2024-09-17T07:50:17.504497+0000","last_undegraded":"2024-09-17T07:50:17.504497+0000","last_fullsized":"2024-09-17T07:50:17.504497+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:32:21.757830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":159,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.506148+0000","last_change":"2024-09-17T07:50:17.505546+0000","last_active":"2024-09-17T07:50:17.506148+0000","last_peered":"2024-09-17T07:50:17.506148+0000","last_clean":"2024-09-17T07:50:17.506148+0000","last_became_active":"2024-09-17T07:50:17.505125+0000","last_became_peered":"2024-09-17T07:50:17.505125+0000","last_unstale":"2024-09-17T07:50:17.506148+0000","last_undegraded":"2024-09-17T07:50:17.506148+0000","last_fullsized":"2024-09-17T07:50:17.506148+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:07:37.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":163,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:50:17.501752+0000","last_change":"2024-09-17T07:50:16.505264+0000","last_active":"2024-09-17T07:50:17.501752+0000","last_peered":"2024-09-17T07:50:17.501752+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:50:16.505264+0000","last_became_peered":"2024-09-17T07:50:16.505264+0000","last_unstale":"2024-09-17T07:50:17.501752+0000","last_undegraded":"2024-09-17T07:50:16.501743+0000","last_fullsized":"2024-09-17T07:50:16.501655+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:40:48.960790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"90'79","reported_seq":248,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.500693+0000","last_change":"2024-09-17T07:48:04.691716+0000","last_active":"2024-09-17T07:50:17.500693+0000","last_peered":"2024-09-17T07:50:17.500693+0000","last_clean":"2024-09-17T07:50:17.500693+0000","last_became_active":"2024-09-17T07:48:04.643230+0000","last_became_peered":"2024-09-17T07:48:04.643230+0000","last_unstale":"2024-09-17T07:50:17.500693+0000","last_undegraded":"2024-09-17T07:50:17.500693+0000","last_fullsized":"2024-09-17T07:50:17.500693+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:16:12.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583080,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":89,"num_write_kb":24610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":33,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.500583+0000","last_change":"2024-09-17T07:50:00.727385+0000","last_active":"2024-09-17T07:50:17.500583+0000","last_peered":"2024-09-17T07:50:17.500583+0000","last_clean":"2024-09-17T07:50:17.500583+0000","last_became_active":"2024-09-17T07:50:00.666799+0000","last_became_peered":"2024-09-17T07:50:00.666799+0000","last_unstale":"2024-09-17T07:50:17.500583+0000","last_undegraded":"2024-09-17T07:50:17.500583+0000","last_fullsized":"2024-09-17T07:50:17.500583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:28:00.373130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027875900000000001,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":162,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-17T07:50:17.507716+0000","last_change":"2024-09-17T07:50:17.506306+0000","last_active":"2024-09-17T07:50:17.507716+0000","last_peered":"2024-09-17T07:50:17.507716+0000","last_clean":"2024-09-17T07:50:17.507716+0000","last_became_active":"2024-09-17T07:50:17.506004+0000","last_became_peered":"2024-09-17T07:50:17.506004+0000","last_unstale":"2024-09-17T07:50:17.507716+0000","last_undegraded":"2024-09-17T07:50:17.507716+0000","last_fullsized":"2024-09-17T07:50:17.507716+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:30:07.923188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":33,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.500455+0000","last_change":"2024-09-17T07:50:00.725975+0000","last_active":"2024-09-17T07:50:17.500455+0000","last_peered":"2024-09-17T07:50:17.500455+0000","last_clean":"2024-09-17T07:50:17.500455+0000","last_became_active":"2024-09-17T07:50:00.666286+0000","last_became_peered":"2024-09-17T07:50:00.666286+0000","last_unstale":"2024-09-17T07:50:17.500455+0000","last_undegraded":"2024-09-17T07:50:17.500455+0000","last_fullsized":"2024-09-17T07:50:17.500455+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:07.875216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":100,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.506413+0000","last_change":"2024-09-17T07:50:17.505770+0000","last_active":"2024-09-17T07:50:17.506413+0000","last_peered":"2024-09-17T07:50:17.506413+0000","last_clean":"2024-09-17T07:50:17.506413+0000","last_became_active":"2024-09-17T07:50:17.505025+0000","last_became_peered":"2024-09-17T07:50:17.505025+0000","last_unstale":"2024-09-17T07:50:17.506413+0000","last_undegraded":"2024-09-17T07:50:17.506413+0000","last_fullsized":"2024-09-17T07:50:17.506413+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:25.481445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":38,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.506085+0000","last_change":"2024-09-17T07:50:17.505407+0000","last_active":"2024-09-17T07:50:17.506085+0000","last_peered":"2024-09-17T07:50:17.506085+0000","last_clean":"2024-09-17T07:50:17.506085+0000","last_became_active":"2024-09-17T07:50:17.505040+0000","last_became_peered":"2024-09-17T07:50:17.505040+0000","last_unstale":"2024-09-17T07:50:17.506085+0000","last_undegraded":"2024-09-17T07:50:17.506085+0000","last_fullsized":"2024-09-17T07:50:17.506085+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:59.055482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":108,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.500597+0000","last_change":"2024-09-17T07:50:00.689800+0000","last_active":"2024-09-17T07:50:17.500597+0000","last_peered":"2024-09-17T07:50:17.500597+0000","last_clean":"2024-09-17T07:50:17.500597+0000","last_became_active":"2024-09-17T07:50:00.666223+0000","last_became_peered":"2024-09-17T07:50:00.666223+0000","last_unstale":"2024-09-17T07:50:17.500597+0000","last_undegraded":"2024-09-17T07:50:17.500597+0000","last_fullsized":"2024-09-17T07:50:17.500597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:05.151332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":8388608,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":115,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:50:17.501757+0000","last_change":"2024-09-17T07:50:16.506409+0000","last_active":"2024-09-17T07:50:17.501757+0000","last_peered":"2024-09-17T07:50:17.501757+0000","last_clean":"2024-09-17T07:50:17.501757+0000","last_became_active":"2024-09-17T07:50:16.505726+0000","last_became_peered":"2024-09-17T07:50:16.505726+0000","last_unstale":"2024-09-17T07:50:17.501757+0000","last_undegraded":"2024-09-17T07:50:17.501757+0000","last_fullsized":"2024-09-17T07:50:17.501757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.888740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":139,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-17T07:50:17.500648+0000","last_change":"2024-09-17T07:50:16.506985+0000","last_active":"2024-09-17T07:50:17.500648+0000","last_peered":"2024-09-17T07:50:17.500648+0000","last_clean":"2024-09-17T07:50:17.500648+0000","last_became_active":"2024-09-17T07:50:16.506641+0000","last_became_peered":"2024-09-17T07:50:16.506641+0000","last_unstale":"2024-09-17T07:50:17.500648+0000","last_undegraded":"2024-09-17T07:50:17.500648+0000","last_fullsized":"2024-09-17T07:50:17.500648+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:18:52.558367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":51,"num_keys_recovered":8,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":16,"num_store_stats":3},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80741058,"num_objects":45,"num_object_clones":1,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3721,"num_read_kb":175579,"num_write":532,"num_write_kb":132278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":25165978,"num_keys_recovered":3,"num_objects_omap":5,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135352320,"data_stored":138413339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":577,"ondisk_log_size":577,"up":6,"acting":7,"num_store_stats":5},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":1,"acting":1,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217915,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":70,"num_read_kb":50,"num_write":44,"num_write_kb":131084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":58720405,"num_keys_recovered":4,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251711488,"data_stored":251658633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":10,"acting":15,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":0,"seq":412316860423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389948,"kb_used_data":84272,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93981892,"statfs":{"total":96636764160,"available":96237457408,"internally_reserved":0,"allocated":86294528,"data_stored":88642303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.55900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.54100000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.497},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54200000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.254}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.31}]}]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319040,"kb_used_data":17516,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94052800,"statfs":{"total":96636764160,"available":96310067200,"internally_reserved":0,"allocated":17936384,"data_stored":17586425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.309}]},{"osd":4,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.24299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640301,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442180,"kb_used_data":136504,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93929660,"statfs":{"total":96636764160,"available":96183971840,"internally_reserved":0,"allocated":139780096,"data_stored":138974034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.30299999999999999,"15min":0.32300000000000001},"min":{"1min":0.20300000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.441,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.29599999999999999,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.35999999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.438,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.62,"5min":0.628,"15min":0.628},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.46400000000000002},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55200000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.5}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.55300000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640302,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422264,"kb_used_data":116588,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93949576,"statfs":{"total":96636764160,"available":96204365824,"internally_reserved":0,"allocated":119386112,"data_stored":118592719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.32000000000000001,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.39000000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.246},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.33800000000000002,"15min":0.35599999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.45100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.26000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46500000000000002,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.504,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.51600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.51400000000000001,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.37}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.313},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301928,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069912,"statfs":{"total":96636764160,"available":96327589888,"internally_reserved":0,"allocated":421888,"data_stored":155934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88092672,"data_stored":88080476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109076480,"data_stored":109052091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16777216,"data_stored":16777216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29376512,"data_stored":29360445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29380608,"data_stored":29360497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":64004096,"data_stored":67109369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:50:26.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.3 2.0 2024-09-17T07:50:26.302 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.302 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:26.346 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:26.345+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:26.409 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:26.407+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:26.449 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.449 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:26.510 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:26.510+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:26.555 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.556 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.564 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:instructing pg(s) [2.0] on osd.1 to force-backfill; instructing pg(s) [3.3] on osd.4 to force-backfill; 2024-09-17T07:50:26.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:50:26.625 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:26.624+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:26.702 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.702 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:26.712 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:26.711+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:50:26.809 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.809 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.813 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:26.813+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:26.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:26.914 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:26.914+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:26.955 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:26.955 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:26.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:26.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:27.051 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:27.050+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:27.071 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:27.071 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:27.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:27.110 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:27.125 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:27.126 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:27.124+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:27.142 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:27.158 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:27.174 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:27.178 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:27.211 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:27.212 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:27.212 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:27.212 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:27.212 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:27.216 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:27.215+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:27.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:27.317 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:27.317+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:27.363 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.197105787, 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:27.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:27.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.18944627100001, 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.369 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:27.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.181056703, 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:27.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:27.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:27.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:27.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.180685453, 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.380 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:27.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.172622488, 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:27.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:27.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:27.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.17225137599999, 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:27.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.393 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.16297097099999, 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:27.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.16131153800001, 2024-09-17T07:50:27.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:27.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.15528671300001, 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:27.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:27.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.15503638600001, 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:27.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:27.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.412 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.147922548, 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:27.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.14740544899999, 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:27.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.128434282, 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.424 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:27.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:27.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:27.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.12217742199999, 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:27.428 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:27.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:27.432 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.11564136000001, 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:27.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:27.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.74277495000001, 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.438 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:27.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:27.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.740814543, 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:27.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:27.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 138.85514590700001, 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.449 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:27.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.54430026899999, 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:27.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:27.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 134.542938824, 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:27.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:27.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.468 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:27.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:27.470 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:27.467+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:27.570 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:27.570+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:27.612 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:27.612 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:27.612 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:27.612 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:27.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:27.671 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:27.671+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:27.721 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:27.722 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:27.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:27.774 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:27.773+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.57594945899999, 2024-09-17T07:50:27.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:27.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:27.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:27.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.207643843, 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:27.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:27.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:27.844 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.51546158799999, 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:27.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:27.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:27.850 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.50960473200001, 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:27.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:27.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:27.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:27.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.497911773, 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.856 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:27.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:27.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.861 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.42905719000001, 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:27.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:27.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:27.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.417157901, 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:27.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:27.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:27.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.366313354, 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:27.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:27.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:27.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:27.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:27.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.36030514699999, 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:27.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.884 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.35460479, 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:27.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.34886516500001, 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:27.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:27.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 135.015754117, 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:27.896 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:27.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:27.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:27.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.17065321999999, 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:27.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.907 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.16456649700001, 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:27.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.081110196, 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.913 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:27.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.07479576399999, 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:27.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.06842430399999, 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:27.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:27.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.06250182799999, 2024-09-17T07:50:27.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:27.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:27.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.004518932, 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:27.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:27.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:27.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 128.04681425999999, 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:27.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:27.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:27.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:27.946 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:27.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:27.948 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:27.946+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:28.048 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:28.048+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:28.056 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:28.056 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:28.056 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:28.057 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:28.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:28.149 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:28.148+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:28.204 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:28.204 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:28.204 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:28.204 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:28.204 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:28.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:28.250 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:28.249+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:28.312 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:28.312 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.14602059800001, 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.314 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.314 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:28.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.314 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:28.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:28.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:28.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:28.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:28.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:28.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:28.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:28.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.322 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.322 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:28.322 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.13868126400001, 2024-09-17T07:50:28.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:28.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:28.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:28.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.13796418999999, 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:28.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:28.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.128736629, 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.332 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.336 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.11951206099999, 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:28.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.10403742099999, 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.345 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.09474791100001, 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.088157666, 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:28.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:28.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.07197518500001, 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.355 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:28.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:28.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:28.359 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.062900265, 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:28.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.72246138899999, 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:28.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:28.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.369 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.720409112, 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:28.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:28.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:28.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:28.376 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.66017746099999, 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:28.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:28.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.498998312, 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:28.382 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.498974011, 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.391 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.498939635, 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:28.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:28.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.49891827799999, 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:28.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.49624963400001, 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.48653206099999, 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:28.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:28.410 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.520421672, 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:28.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:28.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.416 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:28.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:28.418 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:28.417+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:28.515 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:28.515 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:28.515 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:28.515 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:28.518 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:28.518+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:28.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:28.620 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:28.620+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:28.663 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:28.663 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:28.663 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:28.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:28.755 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:28.755+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:28.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:50:28.809 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:28.809 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:28.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.140214907000001, 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:28.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.137202034000001, 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:28.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:28.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.134935109000001, 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:28.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:28.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.132765491000001, 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:28.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:28.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.129476124, 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:28.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.126582866, 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:28.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.123969994999999, 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:28.837 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.12113171, 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.116987870999999, 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:28.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:28.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.102753015000001, 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.092298559, 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.848 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.851 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.087802461999999, 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:28.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.084325290999999, 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:28.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.859 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.064589188999999, 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:28.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:28.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.050048767, 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:28.863 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.046315631999999, 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:28.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:28.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.042553839, 2024-09-17T07:50:28.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:28.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:28.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.035273949, 2024-09-17T07:50:28.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.877 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.020899162999999, 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:28.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.881 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.006943288999999, 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:28.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:28.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:28.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:28.885 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:28.886 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:28.885+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:28.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:28.987 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:28.986+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:28.990 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:28.990 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:28.990 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:28.991 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:28.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:29.087 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:29.086+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:29.100 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:29.100 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:29.101 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:29.101 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:29.101 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:29.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:29.179 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:29.188 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:29.188+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:29.254 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:29.254 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:29.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:29.254 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.087590812, 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.07993129600001, 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:29.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:29.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:29.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:29.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:29.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:29.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:29.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:29.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.071541728, 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:29.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.071170478, 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:29.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.06310751300001, 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:29.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:29.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:29.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:29.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.062736401, 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:29.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.053455996, 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:29.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.05179656300001, 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.04577173800001, 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:29.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.04552141100001, 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:29.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:29.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:29.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:29.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.038407573, 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:29.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:29.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:29.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.03789047399999, 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:29.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.018919307, 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.327 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.012662447, 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.00612638499999, 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.63325997499999, 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.631299568, 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.74563093200001, 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.43478529399999, 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.43342384900001, 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:29.378 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:29.379 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:29.379+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:29.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:29.380 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:29.381 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:29.381 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:29.381 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:29.382 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:29.382 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:29.382 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:29.382 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:29.383 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:29.383 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:29.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:29.480 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:29.479+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:29.525 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:29.525 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:29.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:29.580 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:29.579+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:29.635 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:29.635 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:29.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:29.684 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:29.683+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:29.755 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:50:29.782 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:29.781+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:29.787 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:29.788 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:29.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:29.882 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:29.881+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:29.897 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:29.898 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:29.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:29.983 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:29.982+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:30.003 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.004 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.012 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:30.083 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:30.082+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:30.149 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.149 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.159 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:30.184 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:30.182+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:30.256 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.257 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.265 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:30.284 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:30.284+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:30.362 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.362 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:30.414 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:30.414+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:30.468 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.468 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:30.522 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:30.521+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:30.587 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:30.586+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:30.591 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.591 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:30.687 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:30.687+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:30.745 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.745 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:30.797 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:30.797+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:30.855 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:30.855 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:30.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:30.889 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:30.889+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:30.990 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:30.990+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:31.005 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:31.005 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:31.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:31.091 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:31.090+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:31.115 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:31.115 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:31.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:31.208 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:31.208+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:31.267 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:31.267 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:31.275 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:31.316 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:31.316+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:31.377 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:31.377 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:31.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:31.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:31.429 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:31.429+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:31.484 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:31.484 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:31.486 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:31.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:31.502 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:31.502 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:31.518 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:31.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:31.518 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:31.533 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:31.533+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:31.535 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:31.535 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:31.550 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:31.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:31.584 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:31.584 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:31.585 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-17T07:50:31.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-09-17T07:50:31.595 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:31.594+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.46537700799999, 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:31.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:31.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.458037674, 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:31.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:31.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:31.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.643 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.643 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.4573206, 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:31.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.44809303900001, 2024-09-17T07:50:31.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:31.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.43886847100001, 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:31.653 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:31.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.423393831, 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:31.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:31.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:31.662 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.414104321, 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:31.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.40751407600001, 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.667 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:31.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.671 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.391331595, 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.675 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.38225667500001, 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:31.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:31.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:31.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.041817799, 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:31.680 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:31.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:31.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.03976552200001, 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.691 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 143.979533871, 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.692 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:31.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:31.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:31.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.81835472200001, 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.697 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:31.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:31.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.81833042100001, 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:31.702 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.706 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.81829604500001, 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:31.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:31.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.818274688, 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:31.712 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.81560604399999, 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:31.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:31.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:31.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.805888471, 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:31.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:31.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.83977808200001, 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:31.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:31.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:31.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:31.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:31.733 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:31.733 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:31.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:31.735 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:31.734+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:31.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:50:31.835 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:31.834+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:31.869 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-17T07:50:31.868+0000 7ffb533f1640 -1 mon.a@0(leader).osd e113 definitely_dead 0 2024-09-17T07:50:31.873 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:31.873 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:31.873 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:31.873 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:31.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:31.966 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:31.966+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:32.021 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:32.021 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:32.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:32.072 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:32.071+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:32.137 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:32.137+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:32.170 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:32.170 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:32.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.501158198999999, 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:32.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:32.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.498145325999999, 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:32.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.495878400999999, 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:32.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.493708782999999, 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:32.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.490419416000002, 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.187 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.487526158000001, 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:32.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.484913287000001, 2024-09-17T07:50:32.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:32.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:32.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.482075001999998, 2024-09-17T07:50:32.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:32.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:32.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.477931163000001, 2024-09-17T07:50:32.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:32.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:32.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.463696306999999, 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:32.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.210 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.453241851000001, 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:32.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.448745754000001, 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:32.214 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:32.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.445268583000001, 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:32.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.425532481000001, 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:32.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.410992059000002, 2024-09-17T07:50:32.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:32.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.407258924000001, 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:32.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.403497131000002, 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.236 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.396217240999999, 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:32.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.381842455000001, 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.240 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.243 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.367886581, 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:32.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:32.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:32.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:32.249 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:32.249+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:32.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:32.350 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:32.350+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:32.353 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:32.353 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:32.353 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:32.353 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:32.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:32.451 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:32.450+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:32.463 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:32.463 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:32.463 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:32.463 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:32.463 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:32.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:32.552 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:32.552+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.615 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.448585858, 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:32.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.44092634200001, 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:32.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:32.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:32.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.432536774, 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:32.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:32.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.432165524, 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:32.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.424102559, 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.635 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:32.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.42373144699999, 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:32.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:32.644 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.414451042, 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:32.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.41279160900001, 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.40676678400001, 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:32.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.40651645700001, 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.658 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:32.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:32.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.399402619, 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.39888551999999, 2024-09-17T07:50:32.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:32.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.379914353, 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:32.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:32.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.373657493, 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:32.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:32.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.36712143099999, 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:32.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:32.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:32.689 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.99425502099999, 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:32.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:32.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:32.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:32.695 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.992294614, 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:32.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.10662597800001, 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:32.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:32.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.79578033999999, 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:32.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:32.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:32.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 139.79441889500001, 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:32.713 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:32.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:32.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:32.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:32.719 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:32.720 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:32.720+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:32.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:32.759 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:50:32.763 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.5. 2024-09-17T07:50:32.778 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-17T07:50:32.821 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:32.821+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:32.832 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:32.833 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:32.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:32.925 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:32.925+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:32.983 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:32.983 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:32.983 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:32.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:33.033 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:33.032+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:33.124 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:33.124+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:33.134 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.87597978400001, 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:33.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 145.50767416799999, 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.140 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:33.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.144 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.81549191299999, 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.150 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.80963505700001, 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.79794209799999, 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.729087515, 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:33.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.71718822599999, 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:33.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:33.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.66634367899999, 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.176 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.66033547200001, 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:33.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.65463511499999, 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:33.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.190 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:33.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:33.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:33.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:33.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:33.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:33.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:33.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:33.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:33.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:33.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:33.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:33.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:33.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:33.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:33.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:33.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.203 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.203 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:33.203 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 144.64889549, 2024-09-17T07:50:33.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:33.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:33.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.204 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.31578444199999, 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.47068354499999, 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:33.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.464596822, 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.224 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:33.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:33.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.38114052099999, 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:33.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:33.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.37482608900001, 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:33.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.36845462900001, 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:33.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.36253215299999, 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:33.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 137.30454925699999, 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.259 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:33.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 133.34684458500001, 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:33.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:33.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:33.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:33.270 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:33.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:33.272 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:33.271+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:33.372 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:33.372+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:33.375 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:33.375 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:33.376 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:33.376 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:33.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:33.473 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:33.473+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:33.523 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:33.523 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:33.523 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:33.523 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:33.523 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:33.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:33.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:33.599 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:33.599+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:33.661 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.503472147, 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:33.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.496132813, 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:33.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.49541573900001, 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:33.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:33.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.48618817799999, 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.684 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:33.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:33.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:33.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.47696361000001, 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.690 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.694 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.46148897, 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.698 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.45219946, 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:33.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.44560921499999, 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.703 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.707 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.429426734, 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.42035181400001, 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:33.712 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.07991293800001, 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:33.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:33.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.07786066099999, 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:33.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:33.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.01762901000001, 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:33.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:33.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.733 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.85644986099999, 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:33.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.85642555999999, 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:33.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.743 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.85639118399999, 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:33.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.85636982700001, 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:33.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.853701183, 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:33.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:33.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.84398361000001, 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:33.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:33.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:33.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.87787322099999, 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:33.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:33.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:33.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:33.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:33.769 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:33.770 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:33.771 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:33.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:33.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:33.772 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:33.773 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:33.773 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:33.773 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:33.774 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:33.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:33.775 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:33.770+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:33.871 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:33.871+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:33.885 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:33.885 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:33.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:33.978 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:33.978+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:34.032 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.032 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.041 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:34.082 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:34.082+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:34.175 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:34.174+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:34.178 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.179 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:34.275 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:34.275+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:34.293 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.293 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:34.376 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:34.376+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:34.404 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.404 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:34.497 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:34.497+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:34.556 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.556 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:34.578 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:34.579+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:34.679 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:34.679+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:34.707 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.707 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:34.780 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:34.780+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:34.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:50:34.869 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.869 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:34.881 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:34.881+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:34.982 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:34.981+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:34.985 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:34.985 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:34.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:35.083 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:35.083+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:35.092 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.092 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:35.185 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:35.185+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:35.239 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.239 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:35.289 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:35.289+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:35.385 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.385 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.386 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:35.385+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:35.393 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:35.486 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:35.486+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:35.490 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.490 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:35.587 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:35.587+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:35.595 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.595 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:35.688 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:35.688+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:35.742 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.742 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:35.774 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:50:35.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:35.791 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:35.790+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:35.869 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:35.870 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:35.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:35.889 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:35.889+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:35.903 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:35.919 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:35.919 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:35.919 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:35.935 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:35.935 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:35.965 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:35.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:35.990 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:35.990+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:35.993 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:35.994 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:36.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:36.091 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:36.092+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:36.146 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.979440549, 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.97178103300001, 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:36.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.963391465, 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:36.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:36.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.963020215, 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.164 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.95495725000001, 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:36.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:36.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:36.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:36.173 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.954586138, 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:36.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.945305733, 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.94364630000001, 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.93762147499999, 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.93737114800001, 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.93025731, 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.929740211, 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.91076904400001, 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.904512184, 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.89797612199999, 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.52510971199999, 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.523149305, 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.63748066900001, 2024-09-17T07:50:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 143.326635031, 2024-09-17T07:50:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 143.32527358600001, 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.259 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:36.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:36.261 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:36.260+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:36.361 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:36.361+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:36.404 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:36.404 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:36.404 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:36.404 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:36.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:36.496 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:36.496+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:36.553 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:36.554 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:36.554 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:36.554 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:36.554 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:36.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:36.563 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:36.563+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:36.664 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:36.663+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:36.706 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.44763968699999, 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:36.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:36.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 149.07933407100001, 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:36.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.387151816, 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.716 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:36.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:36.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:36.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:36.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:36.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:36.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.38129495999999, 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:36.730 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.369602001, 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:36.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.732 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:36.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:36.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:36.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.738 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.30074741799999, 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:36.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.288848129, 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:36.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:36.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.238003582, 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:36.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:36.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.231995375, 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:36.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:36.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:36.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:36.762 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.226295018, 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:36.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:36.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.22055539300001, 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.768 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:36.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:36.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 143.88744434500001, 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:36.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:36.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:36.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:36.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.042343448, 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:36.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:36.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 141.03625672499999, 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:36.791 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.952800424, 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:36.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.94648599199999, 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:36.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:36.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.940114532, 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:36.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:36.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.934192056, 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.809 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:36.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:36.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 140.87620916, 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:36.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:36.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.918504488, 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:36.824 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:36.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:36.826 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:36.825+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:36.926 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:36.926+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:36.935 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:36.935 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:36.935 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:36.935 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:36.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:37.028 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:37.028+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:37.083 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:37.084 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:37.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:37.134 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:37.133+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:37.229 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:37.229+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:37.232 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.06540866099999, 2024-09-17T07:50:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.058069327, 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.057352253, 2024-09-17T07:50:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.04812469199999, 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.03890012400001, 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.023425484, 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.014135974, 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.00754572899999, 2024-09-17T07:50:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.991363248, 2024-09-17T07:50:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.98228832800001, 2024-09-17T07:50:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.641849452, 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.63979717500001, 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.579565524, 2024-09-17T07:50:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.41838637500001, 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.41836207399999, 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:37.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:37.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.41832769800001, 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:37.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:37.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:37.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.418306341, 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:37.346 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:37.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.41563769699999, 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.351 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:37.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:37.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.405920124, 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.43980973500001, 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:37.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.367 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:37.369 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:37.369 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:37.368+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:37.469 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:37.469+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:37.509 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:37.509 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:37.509 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:37.509 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:37.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:37.570 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:37.570+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:37.656 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:37.657 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:37.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:37.708 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:37.708+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.093695912999998, 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.763 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.090683040000002, 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:37.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.771 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.088416115000001, 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:37.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:37.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.086246496999998, 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:37.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.082957129999997, 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.080063871999997, 2024-09-17T07:50:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.077451001, 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.074612715999997, 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:37.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:37.798 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.070468877000003, 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:37.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 37.056234021000002, 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:37.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.045779564999997, 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:37.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.810 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.041283468000003, 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:37.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.037806297000003, 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:37.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.018070195, 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:37.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.822 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 32.003529772999997, 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.999796637999999, 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:37.826 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:37.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.996034845000001, 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:37.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.988754955000001, 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:37.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:37.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.837 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.974380169, 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.960424294999999, 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:37.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:37.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:37.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.845 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:37.845 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:37.845 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:37.845 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [0, 3] live_osds: [1, 2, 4, 5] 2024-09-17T07:50:37.845 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-17T07:50:37.846 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T07:50:37.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T07:50:37.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:50:37.847 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:37.846+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:37.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:37.849 INFO:tasks.ceph.osd.0:Started 2024-09-17T07:50:37.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:37.947 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:37.947+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-09-17T07:50:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-09-17T07:50:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:config diff 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:config diff get 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:config get 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:config help [] 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:config set ... 2024-09-17T07:50:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:admin_socket: invalid command 2024-09-17T07:50:37.968 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:37.968 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:37.968 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:37.968 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:50:37.969 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-17T07:50:37.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:37.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:38.048 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:38.048+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:38.123 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:38.124 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:38.124 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:38.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:38.138 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:38.148 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:38.148+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:38.156 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:38.172 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:38.188 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:38.188 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:38.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:38.236 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:38.236 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:38.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:38.289 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:38.288+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:38.347 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:38.347 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:38.350 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:38.351+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:38.355 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:38.451 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:38.451+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:38.465 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:38.464+0000 7ffa3f521440 -1 Falling back to public interface 2024-09-17T07:50:38.497 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:38.497 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:38.506 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:38.590 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:38.590+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:38.648 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:38.648 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:38.652 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:38.652+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:38.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:38.753 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:38.753+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:38.759 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:38.759 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:38.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:38.779 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:50:38.855 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:38.854+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:38.865 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:38.865 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:38.874 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:38.959 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:38.959+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:39.012 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.012 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:39.059 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:39.057+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:39.158 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:39.158+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:39.158 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.159 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:39.259 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:39.259+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:39.305 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.305 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:39.360 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:39.359+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:39.450 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.450 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:39.460 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:39.460+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:39.561 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:39.561+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:39.574 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.574 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:39.663 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:39.662+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:39.685 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.685 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:39.763 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:39.763+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:39.797 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.797 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:39.864 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:39.864+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:39.948 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:39.949 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:39.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:39.965 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:39.965+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:40.059 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:40.059 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:40.066 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:40.066+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:40.068 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:40.167 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:40.167+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:40.169 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:40.169 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:40.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:40.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:40.268 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:40.267+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:40.287 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:40.287 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:40.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:40.334 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:40.335 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:40.350 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:40.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:40.369 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:40.369+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:40.381 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:40.381 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:40.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:40.396 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:40.396 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:40.396 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:40.396 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:40.396 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:40.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:40.470 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:40.470+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.337695978, 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:40.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.33035664400001, 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.509 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:40.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:40.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.32963957000001, 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.513 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:40.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.32041200899999, 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.522 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.31118744099999, 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.29571280100001, 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:40.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.28642329100001, 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:40.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.535 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.27983304599999, 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.539 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.26365056500001, 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:40.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:40.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:40.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:40.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.25457564499999, 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:40.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:40.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.91413676900001, 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:40.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:40.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:40.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.91208449199999, 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:40.555 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:40.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:40.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.85185284100001, 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:40.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:40.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:40.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.69067369199999, 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:40.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.69064939099999, 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:40.577 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.69061501499999, 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:40.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:40.582 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.69059365800001, 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:40.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.587 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.687925014, 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.67820744100001, 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:40.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:40.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.597 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.71209705199999, 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:40.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:40.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:40.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:40.604 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:40.604 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:40.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:40.606 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:40.604+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:40.706 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:40.706+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:40.743 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:40.743 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:40.743 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:40.743 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:40.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:40.793 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:50:40.837 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:40.837+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:40.889 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:40.890 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:40.890 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:40.890 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:40.890 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:40.898 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:40.942 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:40.942+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:40.996 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:40.996 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:40.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.327269866000002, 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:40.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:40.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.324256992999999, 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:41.001 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:41.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:41.005 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.321990067999998, 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.319820450000002, 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:41.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.013 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.316531083000001, 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:41.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:41.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:41.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.313637825000001, 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001046567, 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:41.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.681989+0000", 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3300000000000002e-07 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682454+0000", 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046502300000000001 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682469+0000", 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682475+0000", 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8379999999999996e-06 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682519+0000", 2024-09-17T07:50:41.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3990999999999999e-05 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.682524+0000", 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7090000000000001e-06 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.683035+0000", 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000511106 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.311024953999997, 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:41.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:41.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.308186669000001, 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:41.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.30404283, 2024-09-17T07:50:41.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:41.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.289807973999999, 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.035 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:41.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.279353518000001, 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:41.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:41.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.274857421, 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00097237500000000004, 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720769+0000", 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720770+0000", 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0200000000000001e-07 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720780+0000", 2024-09-17T07:50:41.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.8689999999999999e-06 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720790+0000", 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0715e-05 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720796+0000", 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6890000000000001e-06 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.720833+0000", 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7308999999999998e-05 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.721741+0000", 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090809099999999998 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.046 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.27138025, 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010159710000000001, 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724246+0000", 2024-09-17T07:50:41.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724247+0000", 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.37e-07 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724253+0000", 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5259999999999999e-06 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724263+0000", 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3239999999999992e-06 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724267+0000", 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2400000000000001e-06 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.724309+0000", 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2540999999999998e-05 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.725262+0000", 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000952803 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.251644147999997, 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.050 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:41.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.237103726000001, 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.057 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.233370591000003, 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:41.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.229608798000001, 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098158300000000006, 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766018+0000", 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766019+0000", 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5400000000000001e-06 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766037+0000", 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7710000000000002e-05 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766062+0000", 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4779000000000001e-05 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766072+0000", 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0207000000000001e-05 2024-09-17T07:50:41.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766190+0000", 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117788 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.766999+0000", 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00080955899999999995 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.222328908000001, 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.207954121999997, 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.068 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:41.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:41.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.193998248, 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:41.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:41.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:41.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:41.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.078 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:41.078 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:41.078 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:41.079 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:41.079+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:41.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:41.180 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:41.180+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:41.202 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:41.202 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:41.202 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:41.202 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:41.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:41.281 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:41.280+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:41.318 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:41.319 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:41.319 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:41.319 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:41.319 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:41.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:41.411 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:41.411+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:41.471 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:41.471 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:41.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.30471485699999, 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:41.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.297055341, 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:41.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.28866577299999, 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.28829452299999, 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.280231558, 2024-09-17T07:50:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.27986044599999, 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:41.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:41.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.27058004099999, 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:41.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:41.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:41.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.268920608, 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.505 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.262895783, 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.262645456, 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.513 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:41.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:41.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:41.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.518 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.25553161799999, 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.25501451900001, 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:41.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:41.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.236043352, 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:41.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.22978649199999, 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:41.534 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:41.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.22325043000001, 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:41.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:41.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:41.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:41.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.85038402000001, 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:41.544 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:41.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:41.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.84842361299999, 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:41.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:41.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:41.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:41.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 152.962754977, 2024-09-17T07:50:41.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:41.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:41.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.65190933900001, 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.561 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:41.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:41.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 148.650547894, 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:41.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:41.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.573 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:41.574 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:41.574+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:41.574 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:41.675 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:41.675+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:41.718 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:41.718 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:41.718 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:41.718 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:41.726 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:41.776 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:41.776+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:50:41.870 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:41.871 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:41.871 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:41.871 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:41.871 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:41.877 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:41.877+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:41.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:41.942 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:41.942+0000 7ffa3f521440 -1 osd.0 38 log_to_monitors true 2024-09-17T07:50:41.978 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:41.978+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.72625553500001, 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:41.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:41.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.35794991899999, 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.990 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:41.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:41.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.66576766399999, 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.995 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:41.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:41.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:41.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:42.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.65991080800001, 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:42.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.64821784899999, 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.007 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:42.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.579363266, 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:42.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:42.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:42.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.019 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.56746397699999, 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:42.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:42.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:42.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.51661942999999, 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:42.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:42.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.032 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.51061122300001, 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:42.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:42.038 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.50491086599999, 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:42.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.044 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.499171241, 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:42.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:42.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 149.16606019299999, 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:42.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:42.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.32095929600001, 2024-09-17T07:50:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.314872573, 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.23141627199999, 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.22510184000001, 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.21873038000001, 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.21280790399999, 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.15482500799999, 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:42.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 142.19712033600001, 2024-09-17T07:50:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:42.106 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:42.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:42.108 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:42.108+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:42.209 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:42.208+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:42.212 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:42.212 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:42.212 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:42.213 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:42.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:42.310 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:42.310+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:42.359 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:42.359 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:42.359 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:42.359 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:42.359 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:42.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:42.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:42.429 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:42.429+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:42.489 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-17T07:50:42.505 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:42.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:42.507 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:42.507 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:42.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:42.507 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:42.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.507 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.340756029, 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:42.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:42.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:42.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.33341669500001, 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.514 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:42.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.33269962099999, 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:42.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:42.522 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.32347206, 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.31424749199999, 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.29877285200001, 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.532 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:42.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.28948334200001, 2024-09-17T07:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:42.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.541 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.282893097, 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:42.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:42.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.26671061600001, 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:42.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:42.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.25763569599999, 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:42.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.91719681999999, 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:42.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:42.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.915144543, 2024-09-17T07:50:42.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:42.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:42.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.85491289199999, 2024-09-17T07:50:42.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:42.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.693733743, 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.693709442, 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.693675066, 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.69365370899999, 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:42.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:42.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:42.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.596 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.69098506500001, 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:42.597 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.603 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.68126749199999, 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:42.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:42.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:42.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.715157103, 2024-09-17T07:50:42.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:42.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:42.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:42.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.617 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:42.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:42.618 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:42.619 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:42.621 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:42.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:42.623 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:42.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:42.625 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:42.619+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:42.720 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:42.719+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:42.723 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:42.723 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:42.732 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:42.821 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:42.821+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:42.833 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:42.833+0000 7ffa32fe8640 -1 osd.0 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:50:42.870 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:42.870 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:42.879 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:42.922 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:42.922+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:42.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:42.977 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:42.977 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:42.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:43.059 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:43.059+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:43.076 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:43.077 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:43.085 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-17T07:50:43.086 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T07:50:43.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T07:50:43.088 INFO:tasks.ceph.osd.3:Started 2024-09-17T07:50:43.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.118 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:43.131 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:43.131+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:no valid command found; 10 closest matches: 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:0 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:1 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:2 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:abort 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:assert 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:config diff 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:config diff get 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:config get 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:config help [] 2024-09-17T07:50:43.181 INFO:teuthology.orchestra.run.smithi094.stderr:config set ... 2024-09-17T07:50:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: invalid command 2024-09-17T07:50:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:50:43.183 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-17T07:50:43.226 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:43.225+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:43.269 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.269 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:43.362 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:43.362+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:43.420 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.420 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.427 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:43.426+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:43.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:43.528 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:43.528+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:43.570 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.570 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:43.628 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:43.628+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:43.673 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:43.671+0000 7f4ba5c22440 -1 Falling back to public interface 2024-09-17T07:50:43.679 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.679 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:43.729 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:43.728+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:43.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:50:43.834 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.835 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.835 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:43.834+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:43.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:43.931 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:43.929+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:43.943 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:43.943 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:43.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:44.032 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:44.031+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:44.088 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:44.088 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:44.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:44.133 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:44.132+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:44.194 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:44.194 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:44.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:44.246 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:44.245+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:44.299 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:44.324 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:44.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:44.335 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:44.334+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:44.436 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:44.435+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:44.463 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:44.463 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:44.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:44.557 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:44.556+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:44.611 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:44.611 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:44.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:44.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:44.660 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:44.659+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:44.736 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:44.736 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:44.736 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:44.737 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:44.739 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:44.738+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:44.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:44.787 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:44.787 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-17T07:50:44.798 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:50:44.811 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:44.811 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:44.827 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:44.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:44.827 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:44.839 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:44.838+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:44.842 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:44.843 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:44.858 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:44.858 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:44.862 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:44.896 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:44.896 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:44.896 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:44.897 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:44.897 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:44.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:44.940 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:44.939+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:45.041 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:45.041+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5371.0:147 1.0 1.76a76788 (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e121)", 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:43.795123+0000", 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.2511804149999999, 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00187846, 2024-09-17T07:50:45.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795123+0000", 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795123+0000", 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795124+0000", 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.364e-06 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795131+0000", 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9770000000000003e-06 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795144+0000", 2024-09-17T07:50:45.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2454e-05 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795152+0000", 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3059999999999998e-06 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795217+0000", 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5543000000000006e-05 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795224+0000", 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.827e-06 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.796979+0000", 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017545900000000001 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797001+0000", 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2399e-05 2024-09-17T07:50:45.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:77 1.0 1.76a76788 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e121)", 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:43.795168+0000", 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.2511347740000001, 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018648060000000001, 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795168+0000", 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795168+0000", 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795169+0000", 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9699999999999996e-07 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795172+0000", 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4369999999999999e-06 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795180+0000", 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8140000000000001e-06 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.795184+0000", 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2719999999999999e-06 2024-09-17T07:50:45.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797027+0000", 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018425360000000001 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797033+0000", 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000003e-06 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.054 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:78 1.0 1.76a76788 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e121)", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:43.796991+0000", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.2493121659999999, 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8894999999999997e-05, 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.796991+0000", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.796991+0000", 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.796992+0000", 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.238e-06 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.796997+0000", 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9470000000000002e-06 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797011+0000", 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4295e-05 2024-09-17T07:50:45.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797019+0000", 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3069999999999997e-06 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797054+0000", 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5729000000000003e-05 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797060+0000", 2024-09-17T07:50:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3789999999999997e-06 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_backoff(1.0 ack-block id 1 [1:00000000::::0,1:20000000::::head) e121)", 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:43.797364+0000", 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.248938965, 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8510999999999994e-05, 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797364+0000", 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797364+0000", 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797365+0000", 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4300000000000002e-07 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797369+0000", 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.087e-06 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797380+0000", 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0733e-05 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797388+0000", 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4149999999999999e-06 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797431+0000", 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3050000000000003e-05 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:43.797443+0000", 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1382999999999999e-05 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.062 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:45.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:45.142 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:45.141+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:45.204 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.204 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:45.204 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:45.204 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:45.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:45.242 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:45.241+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:45.313 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.314 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:45.314 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:45.314 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:45.314 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:45.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:45.366 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:45.366+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.25821607, 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.25055655400001, 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.242166986, 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.241795736, 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.23373277100001, 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.233361659, 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.224081254, 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.22242182100001, 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.21639699599999, 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.21614666900001, 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.209032831, 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.208515732, 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.18954456500001, 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.183287705, 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.17675164299999, 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.80388523299999, 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:45.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.801924826, 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 156.91625619000001, 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 152.605410552, 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 152.60404910700001, 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.532 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:45.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:45.534 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:45.533+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:45.634 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:45.633+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:45.675 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.675 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:45.675 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:45.675 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:45.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:45.735 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:45.735+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:45.785 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:45.786 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:45.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:45.836 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:45.835+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:45.936 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:45.935+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.69812472199999, 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:45.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:45.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.329819106, 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:45.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:45.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.637636851, 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:45.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.63177999499999, 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.972 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:45.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:45.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.620087036, 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.978 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.55123245300001, 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.539333164, 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:45.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:45.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.488488617, 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:45.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:45.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.48248040999999, 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.001 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:46.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:46.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.476780053, 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.008 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:46.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.47104042800001, 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:46.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:46.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 153.13792938, 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.021 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:46.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.29282848299999, 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.027 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.28674176000001, 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.033 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:46.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.038 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.203285459, 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:46.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.19697102699999, 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:46.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:46.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:46.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.19059956699999, 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:46.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.184677091, 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:46.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:46.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.126694195, 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.062 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 146.16898952299999, 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:46.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:46.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:46.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:46.072 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:46.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:46.074 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:46.073+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:46.173 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:46.173 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:46.173 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:46.173 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:46.174 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:46.173+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:46.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:46.275 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:46.274+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:46.278 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:46.279 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:46.287 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:46.376 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:46.375+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.22314219899999, 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.390 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:46.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:46.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.215802865, 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:46.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.21508579100001, 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:46.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:46.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.405 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.20585822999999, 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:46.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:46.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.19663366200001, 2024-09-17T07:50:46.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:46.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.181159022, 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.415 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:46.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.171869512, 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:46.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:46.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.16527926699999, 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:46.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:46.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.428 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.149096786, 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:46.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:46.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.14002186600001, 2024-09-17T07:50:46.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:46.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.79958299, 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:46.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:46.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.79753071299999, 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.73729906200001, 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.57611991300001, 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.57609561199999, 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.57606123599999, 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.57603987900001, 2024-09-17T07:50:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.573371235, 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.56365366200001, 2024-09-17T07:50:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.59754327300001, 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.492 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:46.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:46.494 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:46.493+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:46.591 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:46.591 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:46.591 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:46.594 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:46.594+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:46.599 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:46.695 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:46.694+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:46.696 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:46.696 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:46.696 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:46.696 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:46.696 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:46.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:46.797 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:46.796+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:46.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.174580433999999, 2024-09-17T07:50:46.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:46.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:46.848 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.171567561000003, 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:46.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.169300636000003, 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:46.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.167131017999999, 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:46.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:46.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.163841650999998, 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:46.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.158335522000002, 2024-09-17T07:50:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.155497236999999, 2024-09-17T07:50:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:46.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.151353397999998, 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:46.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.137118542000003, 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.883 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:46.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.126664085999998, 2024-09-17T07:50:46.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:46.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.098954716000001, 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.084414293999998, 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.895 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:46.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.080681159000001, 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:46.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:46.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.069639475999999, 2024-09-17T07:50:46.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:46.907 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.055264690000001, 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:46.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.041308815999997, 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.911 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:46.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.914 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.037674733, 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 1.032668873, 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:50:46.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.922 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.96538871100000001, 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.956464643, 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:46.926 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:46.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:46.930 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:46.930 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:46.932 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:46.932 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:46.930+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:47.033 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:47.031+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:47.039 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:47.039 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:47.039 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:47.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:47.052 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:47.053 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:50:47.070 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:47.086 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:47.086 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:47.086 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-17T07:50:47.103 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:47.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:47.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:47.118 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:50:47.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:50:47.151 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.151 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.151 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:47.150+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:47.159 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:47.161 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:47.160+0000 7f4ba5c22440 -1 osd.3 49 log_to_monitors true 2024-09-17T07:50:47.237 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:47.236+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:47.301 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.301 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:47.352 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:47.352+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.412 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:47.440 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:47.437+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:47.507 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:50:47.539 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:47.538+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:47.549 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:47.548+0000 7f4b993e8640 -1 osd.3 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:50:47.566 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.566 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.574 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:47.640 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:47.639+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:47.676 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.676 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:47.742 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:47.741+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:47.787 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.787 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:47.879 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:47.879+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:47.937 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:47.937 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:47.943 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:47.942+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:47.945 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:48.044 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:48.044+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:48.087 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.087 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:48.145 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:48.145+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:48.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:48.199 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.199 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:48.246 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:48.246+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:48.285 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:48.286 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:48.286 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:48.286 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:48.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-17T07:50:48.348 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:48.346+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:48.348 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.348 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:48.448 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:48.447+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:48.493 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.493 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:48.585 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:48.584+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:48.639 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.639 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.647 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:48.688 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:48.687+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:48.751 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:48.750+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:48.785 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.786 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:48.852 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:48.851+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:48.892 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:48.892 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:48.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:48.952 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:48.952+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:49.005 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:49.005 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:49.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:49.056 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:49.056+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:49.115 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:49.115 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:49.123 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:49.123 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:49.165 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:49.164+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:49.255 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:49.254+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:49.268 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:49.268 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:49.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:49.281 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:49.282 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:50:49.297 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:49.297 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:49.312 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:49.312 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:49.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:50:49.328 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-17T07:50:49.343 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:49.356 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:49.356+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:49.358 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:49.358 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:49.376 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:49.376 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:49.376 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:49.376 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:49.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:50:49.457 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:49.456+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:49.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:50:49.509 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:50:49.517 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 100% complete...done. 2024-09-17T07:50:49.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:50:49.523 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:50:49.523 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:50:49.523 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:50:49.524 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:50:49.524 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:50:49.527 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:50:49.527 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:50:49.527 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:50:49.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-17T07:50:49.550 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:50:49.550 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:50:49.550 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115593 s, 90.7 MB/s 2024-09-17T07:50:49.551 INFO:tasks.workunit.client.0.smithi098.stderr:++ cat /tmp/tmp.TyfTMF4aW5/quiesce 2024-09-17T07:50:49.552 INFO:tasks.workunit.client.0.smithi098.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-17T07:50:49.553 INFO:tasks.workunit.client.0.smithi098.stderr:++ cat /tmp/tmp.TyfTMF4aW5/unquiesce 2024-09-17T07:50:49.553 INFO:tasks.workunit.client.0.smithi098.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-17T07:50:49.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ touch /tmp/tmp.TyfTMF4aW5/quiesce.sh 2024-09-17T07:50:49.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod +x /tmp/tmp.TyfTMF4aW5/quiesce.sh 2024-09-17T07:50:49.555 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-09-17T07:50:49.556 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false rbd snap create rbd/testrbdnbd27356@quiesce2 2024-09-17T07:50:49.556 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/testrbdnbd27356@quiesce2 2024-09-17T07:50:49.568 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:49.568+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:49.604 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-09-17T07:50:49.604 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-09-17T07:50:49.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:50:49.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:50:49.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:50:49.609 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:50:49.610 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:50:49.610 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:50:49.612 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:50:49.612 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:50:49.612 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:50:49.629 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:50:49.630 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:50:49.630 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00732079 s, 143 MB/s 2024-09-17T07:50:49.631 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-09-17T07:50:49.632 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/testrbdnbd27356@quiesce2 2024-09-17T07:50:49.635 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:49.636 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-09-17T07:50:49.636 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:49.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:49.689 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:49.689+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:49.760 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:49.760+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:49.766 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:49.766 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:49.766 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:49.766 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:49.775 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:49.861 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:49.861+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:49.917 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:49.918 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:49.918 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:49.918 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:49.918 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:49.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:49.962 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:49.961+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:50.063 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:50.062+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:50.068 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:50.068 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:50.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.90189988099999, 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.894240365, 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.073 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:50.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.88585079699999, 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:50.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:50.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.88547954699999, 2024-09-17T07:50:50.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:50.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:50.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.877416582, 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:50.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:50.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.87704547000001, 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:50.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.86776506499999, 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:50.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.866105632, 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:50.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:50.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.860080807, 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:50.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.85983048, 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.113 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:50.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:50.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.85271664199999, 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:50.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:50.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.85219954300001, 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:50.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:50.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:50.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:50.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.83322837599999, 2024-09-17T07:50:50.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:50.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:50.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.82697151599999, 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:50.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:50.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.82043545400001, 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:50.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:50.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.143 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.44756904400001, 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:50.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:50.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.44560863699999, 2024-09-17T07:50:50.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:50.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:50.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:50.155 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 161.559940001, 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:50.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:50.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:50.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.24909436300001, 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:50.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:50.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:50.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.247732918, 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.173 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:50.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:50.175 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:50.174+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:50.275 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:50.275+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:50.317 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:50.317 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:50.317 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:50.317 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:50.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:50.376 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:50.376+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:50.467 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:50.467 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:50.467 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:50.467 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:50.467 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:50.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:50.476 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:50.475+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:50:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:50:50.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-17T07:50:50.577 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:50.576+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:50.589 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 168.33034851799999, 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:50.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:50.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.96204290200001, 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:50.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.598 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.269860647, 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:50.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:50.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.26400379099999, 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.605 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:50.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:50.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.25231083200001, 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.611 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:50.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.18345624899999, 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:50.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:50.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.17155696, 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.623 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:50.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.628 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.12071241300001, 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:50.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.114704206, 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.109003849, 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.10326422399999, 2024-09-17T07:50:50.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.77015317600001, 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.925052279, 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:50.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:50.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.91896555599999, 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:50.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.83550925500001, 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.670 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:50.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.82919482299999, 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:50.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:50.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:50.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.822823363, 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.682 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:50.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:50.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.816900887, 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.688 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:50.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:50.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:50.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.758917991, 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:50.694 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.801213319, 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:50.704 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:50.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:50.706 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:50.704+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:50.805 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:50.804+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:50.809 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:50.810 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:50.810 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:50.810 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:50.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:50.907 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:50.905+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:50.956 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:50.956 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:50.956 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:50.956 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:50.956 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:50.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:51.007 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:51.007+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.937260447, 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:51.104 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:51.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.92992111300001, 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.113 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.92920403900001, 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:51.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.919976478, 2024-09-17T07:50:51.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:51.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:51.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.91075190999999, 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.129 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.89527727000001, 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.130 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:51.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.134 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.88598776000001, 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:51.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:51.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:51.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.87939751499999, 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:51.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.143 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.86321503400001, 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:51.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:51.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 168.85414011399999, 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:51.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:51.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:51.152 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 163.51370123800001, 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:51.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:51.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:51.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 163.51164896099999, 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 163.45141731000001, 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:51.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:51.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:51.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.29023816099999, 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:51.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.174 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.29021385999999, 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:51.175 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:51.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.29017948399999, 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.180 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:51.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.29015812700001, 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.185 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:51.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.287489483, 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:51.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 158.27777191000001, 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.195 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:51.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.31166152099999, 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:51.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.205 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:51.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:51.207 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:51.206+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:51.304 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:51.304 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:51.304 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:51.307 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:51.306+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:51.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:51.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:51.409 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:51.409 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:51.409 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:51.409 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:51.410 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:51.414 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:51.414+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:51.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:51.493 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:51.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:50:51.509 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:51.508+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:51.509 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:51.509 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:51.517 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:51.517 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:51.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:51.517 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:51.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.848031122000002, 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:51.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:51.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.521 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.845018248999999, 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.525 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.842751323999998, 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:51.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.840581706000002, 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:51.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.837292339000001, 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:51.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.537 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.831786209999997, 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.828947925000001, 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.824804086, 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:51.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.810569229999999, 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.800114774000001, 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.772405403999997, 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.757864982000001, 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:51.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.566 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.754131846999996, 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:51.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.743090164000002, 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.570 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:51.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:51.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.728715377999997, 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:51.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.577 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.714759504, 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:51.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.7111254210000002, 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.7061195609999995, 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:51.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:51.588 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6388393990000001, 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:50:51.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.6299153310000003, 2024-09-17T07:50:51.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.595 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:51.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:51.596 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T07:50:51.597 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:51.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:51.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-17T07:50:51.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:50:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:50:51.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:51.600 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:51.600 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:51.639 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:51.638+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:51.711 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:51.711+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:51.742 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:51.742 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:51.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:51.812 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:51.811+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:51.854 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:51.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:51.913 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:51.912+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:52.003 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.003 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.012 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:52.013 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:52.012+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:52.112 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.112 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.114 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:52.113+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:52.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:50:52.215 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:52.214+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:52.258 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.258 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:50:52.316 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:52.316+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:52.404 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.404 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:52.456 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:52.456+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:52.517 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:52.517+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:52.520 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.520 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.524 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:50:52.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:50:52.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:52.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-17T07:50:52.619 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:52.617+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:52.711 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.711 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.719 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:52.718+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:52.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:52.820 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:52.819+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:52.826 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.826 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:52.921 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:52.920+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:52.945 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:53.022 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:53.021+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:53.046 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.046 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:53.124 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:53.123+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:53.155 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.155 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.164 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:53.225 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:53.224+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:53.271 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.271 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:53.325 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:53.325+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:53.378 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.378 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:53.426 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:53.425+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:53.485 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.485 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:53.526 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:53.525+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:53.527 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:50:53.527 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:50:53.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-17T07:50:53.595 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.595 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:53.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:53.647 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:53.646+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:53.701 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:53.701 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:53.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:53.728 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:53.727+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:53.759 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:53.759 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:50:53.775 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:53.775 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:53.790 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:53.806 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:50:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:50:53.821 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:53.829 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:53.828+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:53.836 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:53.836 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:53.850 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:53.850 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:53.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:53.850 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:53.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.850 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.180676962, 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:53.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.177664088999997, 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:53.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.858 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.175397164000003, 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:53.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:53.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.173227546, 2024-09-17T07:50:53.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:53.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.169938178999999, 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:53.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.164432050000002, 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.161593764999999, 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:53.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.157449925999998, 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:53.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:53.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.143215069999997, 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:53.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:53.886 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.132760613999999, 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:53.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.105051244000002, 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:53.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.892 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:53.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.090510821999999, 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.899 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.086777687000001, 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:53.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.903 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.075736003999999, 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:53.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.061361218000002, 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.907 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:53.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.047405343999998, 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:53.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:53.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:53.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.0437712609999998, 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:50:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 8.0387654009999991, 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:53.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.9714852389999997, 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:50:53.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.9625611709999999, 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:50:53.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:50:53.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:53.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:53.930 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:53.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:53.932 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:53.931+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:54.032 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:54.032+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:54.033 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:54.033 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:54.033 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:54.033 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:54.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:54.133 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:54.133+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:54.183 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:54.184 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:54.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:54.234 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:54.233+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:54.335 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:54.334+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.5292930509999998, 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.336 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:50:54.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.5278000279999997, 2024-09-17T07:50:54.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:50:54.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:50:54.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:50:54.344 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.5228144449999998, 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:50:54.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.5186749220000006, 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:50:54.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:50:54.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.5138906009999999, 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:50:54.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:50:54.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.5061193539999991, 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:50:54.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4960811130000007, 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.362 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:50:54.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4767635529999996, 2024-09-17T07:50:54.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4643652449999998, 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:50:54.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4607136890000003, 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:50:54.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4601305379999996, 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.379 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:50:54.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:50:54.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4555471959999995, 2024-09-17T07:50:54.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.387 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4552181449999999, 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:54.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4470189490000003, 2024-09-17T07:50:54.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.395 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4464977020000003, 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:50:54.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.8162108950000002, 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:50:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7918610939999997, 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:50:54.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:54.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.408 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7903342850000001, 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.412 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7590892790000003, 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:50:54.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:50:54.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7072662200000002, 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:50:54.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:50:54.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:54.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:54.422 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:54.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:54.465 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:54.464+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:54.523 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:54.524 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:54.524 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:54.524 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:54.530 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:50:54.533 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:50:54.543 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:50:54.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:50:54.576 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:54.575+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:54.630 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:54.630 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:54.630 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:54.630 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:54.630 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:54.638 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:54.636+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:54.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:50:54.739 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:54.738+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:54.777 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:54.777 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2594117779999996, 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:50:54.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2437921120000004, 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:50:54.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2397112899999998, 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:50:54.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.789 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2332211480000002, 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:50:54.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2275059380000002, 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:50:54.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2202683030000001, 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:50:54.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:50:54.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.800 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2127376979999998, 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:50:54.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:50:54.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:50:54.804 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2089624639999998, 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:50:54.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2575956579999996, 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:50:54.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2528867999999997, 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:50:54.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:50:54.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2489897479999996, 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:50:54.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.821 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2454005720000003, 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:50:54.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2409162550000001, 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:50:54.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2369842249999996, 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:50:54.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2331973859999996, 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:50:54.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2295457040000004, 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:50:54.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:50:54.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2254372629999999, 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.842 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:50:54.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2218371919999997, 2024-09-17T07:50:54.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.850 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2184705989999998, 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:50:54.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:50:54.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 4.2150788649999997, 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:50:54.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:50:54.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:50:54.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.859 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:54.859 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:54.859 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:54.860 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:50:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:50:54.862 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:54.859+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:54.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:54.871 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-17T07:50:54.529596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216598063,"num_objects":101,"num_object_clones":2,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":6,"num_read":4158,"num_read_kb":177929,"num_write":699,"num_write_kb":267050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":272630673,"num_keys_recovered":47,"num_objects_omap":12,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":728,"ondisk_log_size":728,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2230208,"kb_used_data":418980,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564000832,"statfs":{"total":579820584960,"available":577536851968,"internally_reserved":0,"allocated":429035520,"data_stored":433022114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17825868,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":295,"num_read_kb":2238,"num_write":22,"num_write_kb":18440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":37748904,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019227"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":42,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.528482+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:50:52.528482+0000","last_peered":"2024-09-17T07:50:52.528482+0000","last_clean":"2024-09-17T07:50:52.528482+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:50:52.528482+0000","last_undegraded":"2024-09-17T07:50:52.528482+0000","last_fullsized":"2024-09-17T07:50:52.528482+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"127'8","reported_seq":44,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521535+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:50:51.521535+0000","last_peered":"2024-09-17T07:50:51.521535+0000","last_clean":"2024-09-17T07:50:51.521535+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:50:51.521535+0000","last_undegraded":"2024-09-17T07:50:51.521535+0000","last_fullsized":"2024-09-17T07:50:51.521535+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":7,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665207+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:50:50.665207+0000","last_peered":"2024-09-17T07:50:50.665207+0000","last_clean":"2024-09-17T07:50:50.665207+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:50:50.665207+0000","last_undegraded":"2024-09-17T07:50:50.665207+0000","last_fullsized":"2024-09-17T07:50:50.665207+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.183389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":820,"num_read_kb":28457,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":96,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665274+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:50:50.665274+0000","last_peered":"2024-09-17T07:50:50.665274+0000","last_clean":"2024-09-17T07:50:50.665274+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:50:50.665274+0000","last_undegraded":"2024-09-17T07:50:50.665274+0000","last_fullsized":"2024-09-17T07:50:50.665274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":112,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525822+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:50:52.525822+0000","last_peered":"2024-09-17T07:50:52.525822+0000","last_clean":"2024-09-17T07:50:52.525822+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:50:52.525822+0000","last_undegraded":"2024-09-17T07:50:52.525822+0000","last_fullsized":"2024-09-17T07:50:52.525822+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"127'270","reported_seq":1348,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.512071+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:50:50.512071+0000","last_peered":"2024-09-17T07:50:50.512071+0000","last_clean":"2024-09-17T07:50:50.512071+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:50:50.512071+0000","last_undegraded":"2024-09-17T07:50:50.512071+0000","last_fullsized":"2024-09-17T07:50:50.512071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39846109,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":1396,"num_read_kb":78369,"num_write":319,"num_write_kb":60555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":123,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665263+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:50:50.665263+0000","last_peered":"2024-09-17T07:50:50.665263+0000","last_clean":"2024-09-17T07:50:50.665263+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:50:50.665263+0000","last_undegraded":"2024-09-17T07:50:50.665263+0000","last_fullsized":"2024-09-17T07:50:50.665263+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"45'5","reported_seq":36,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T07:50:46.501809+0000","last_change":"2024-09-17T07:50:45.891597+0000","last_active":"2024-09-17T07:50:46.501809+0000","last_peered":"2024-09-17T07:50:46.501809+0000","last_clean":"2024-09-17T07:50:46.501809+0000","last_became_active":"2024-09-17T07:50:45.801562+0000","last_became_peered":"2024-09-17T07:50:45.801562+0000","last_unstale":"2024-09-17T07:50:46.501809+0000","last_undegraded":"2024-09-17T07:50:46.501809+0000","last_fullsized":"2024-09-17T07:50:46.501809+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:45.245789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'206","reported_seq":1105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.515384+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:50:51.515384+0000","last_peered":"2024-09-17T07:50:51.515384+0000","last_clean":"2024-09-17T07:50:51.515384+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:50:51.515384+0000","last_undegraded":"2024-09-17T07:50:51.515384+0000","last_fullsized":"2024-09-17T07:50:51.515384+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:00.953341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777455,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1556,"num_read_kb":46303,"num_write":103,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521586+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:50:51.521586+0000","last_peered":"2024-09-17T07:50:51.521586+0000","last_clean":"2024-09-17T07:50:51.521586+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:50:51.521586+0000","last_undegraded":"2024-09-17T07:50:51.521586+0000","last_fullsized":"2024-09-17T07:50:51.521586+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":212,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665226+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:50:50.665226+0000","last_peered":"2024-09-17T07:50:50.665226+0000","last_clean":"2024-09-17T07:50:50.665226+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:50:50.665226+0000","last_undegraded":"2024-09-17T07:50:50.665226+0000","last_fullsized":"2024-09-17T07:50:50.665226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":190,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665335+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:50:50.665335+0000","last_peered":"2024-09-17T07:50:50.665335+0000","last_clean":"2024-09-17T07:50:50.665335+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:50:50.665335+0000","last_undegraded":"2024-09-17T07:50:50.665335+0000","last_fullsized":"2024-09-17T07:50:50.665335+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"90'79","reported_seq":294,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.568181+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:50:50.568181+0000","last_peered":"2024-09-17T07:50:50.568181+0000","last_clean":"2024-09-17T07:50:50.568181+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:50:50.568181+0000","last_undegraded":"2024-09-17T07:50:50.568181+0000","last_fullsized":"2024-09-17T07:50:50.568181+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583080,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":89,"num_write_kb":24610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":54,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-17T07:50:46.055241+0000","last_change":"2024-09-17T07:50:46.054762+0000","last_active":"2024-09-17T07:50:46.055241+0000","last_peered":"2024-09-17T07:50:46.055241+0000","last_clean":"2024-09-17T07:50:46.055241+0000","last_became_active":"2024-09-17T07:50:45.803389+0000","last_became_peered":"2024-09-17T07:50:45.803389+0000","last_unstale":"2024-09-17T07:50:46.055241+0000","last_undegraded":"2024-09-17T07:50:46.055241+0000","last_fullsized":"2024-09-17T07:50:46.055241+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:46:23.290071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":205,"reported_epoch":129,"state":"peering","last_fresh":"2024-09-17T07:50:51.520412+0000","last_change":"2024-09-17T07:50:51.520678+0000","last_active":"2024-09-17T07:50:51.520377+0000","last_peered":"2024-09-17T07:50:50.665211+0000","last_clean":"2024-09-17T07:50:50.665211+0000","last_became_active":"2024-09-17T07:50:33.766392+0000","last_became_peered":"2024-09-17T07:50:33.766392+0000","last_unstale":"2024-09-17T07:50:51.520412+0000","last_undegraded":"2024-09-17T07:50:51.520412+0000","last_fullsized":"2024-09-17T07:50:51.520412+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:36.039448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":33554432,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":26,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.512039+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:50:50.512039+0000","last_peered":"2024-09-17T07:50:50.512039+0000","last_clean":"2024-09-17T07:50:50.512039+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:50:50.512039+0000","last_undegraded":"2024-09-17T07:50:50.512039+0000","last_fullsized":"2024-09-17T07:50:50.512039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665244+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:50:50.665244+0000","last_peered":"2024-09-17T07:50:50.665244+0000","last_clean":"2024-09-17T07:50:50.665244+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:50:50.665244+0000","last_undegraded":"2024-09-17T07:50:50.665244+0000","last_fullsized":"2024-09-17T07:50:50.665244+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":22,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521583+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:50:51.521583+0000","last_peered":"2024-09-17T07:50:51.521583+0000","last_clean":"2024-09-17T07:50:51.521583+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:50:51.521583+0000","last_undegraded":"2024-09-17T07:50:51.521583+0000","last_fullsized":"2024-09-17T07:50:51.521583+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":41,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.556404+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:50:52.556404+0000","last_peered":"2024-09-17T07:50:52.556404+0000","last_clean":"2024-09-17T07:50:52.556404+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:50:52.556404+0000","last_undegraded":"2024-09-17T07:50:52.556404+0000","last_fullsized":"2024-09-17T07:50:52.556404+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665342+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:50:50.665342+0000","last_peered":"2024-09-17T07:50:50.665342+0000","last_clean":"2024-09-17T07:50:50.665342+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:50:50.665342+0000","last_undegraded":"2024-09-17T07:50:50.665342+0000","last_fullsized":"2024-09-17T07:50:50.665342+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":35,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525665+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:50:52.525665+0000","last_peered":"2024-09-17T07:50:52.525665+0000","last_clean":"2024-09-17T07:50:52.525665+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:50:52.525665+0000","last_undegraded":"2024-09-17T07:50:52.525665+0000","last_fullsized":"2024-09-17T07:50:52.525665+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81789672,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3986,"num_read_kb":177795,"num_write":547,"num_write_kb":134331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":5,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166830080,"data_stored":176162256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217953,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":45,"num_write_kb":131085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":180355259,"num_keys_recovered":25,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251715584,"data_stored":251658690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860429,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369340,"kb_used_data":71208,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002500,"statfs":{"total":96636764160,"available":96258560000,"internally_reserved":0,"allocated":72916992,"data_stored":71985413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640307,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354300,"kb_used_data":48624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94017540,"statfs":{"total":96636764160,"available":96273960960,"internally_reserved":0,"allocated":49790976,"data_stored":55191646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.55900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54200000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39600000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.33500000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.27500000000000002}]}]},{"osd":3,"up_from":126,"seq":541165879298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328716,"kb_used_data":30780,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94043124,"statfs":{"total":96636764160,"available":96300158976,"internally_reserved":0,"allocated":31518720,"data_stored":30621693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360460,"kb_used_data":54784,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94011380,"statfs":{"total":96636764160,"available":96267653120,"internally_reserved":0,"allocated":56098816,"data_stored":55191905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.30299999999999999,"15min":0.32300000000000001},"min":{"1min":0.20300000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.441,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.29599999999999999,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.438,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.62,"5min":0.628,"15min":0.628},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.46400000000000002},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55200000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38100000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.373},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640308,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410192,"kb_used_data":104508,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93961648,"statfs":{"total":96636764160,"available":96216727552,"internally_reserved":0,"allocated":107016192,"data_stored":106113853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.32000000000000001,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.39000000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.33800000000000002,"15min":0.35599999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.45100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.32600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46500000000000002,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.504,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.51600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.51400000000000001,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53100000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48999999999999999}]}]},{"osd":0,"up_from":121,"seq":519691042819,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407200,"kb_used_data":109076,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93964640,"statfs":{"total":96636764160,"available":96219791360,"internally_reserved":0,"allocated":111693824,"data_stored":113917604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":67108997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16781312,"data_stored":16777254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43012096,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29380608,"data_stored":29360483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12583080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39866368,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:50:54.874 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:50:54.961 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:54.960+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:54.965 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:54.965 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:54.965 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:54.965 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:54.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:55.061 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:55.061+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:55.094 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:55.094 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:55.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:55.162 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:55.161+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:50:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:50:55.220 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-17T07:50:54.529596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216598063,"num_objects":101,"num_object_clones":2,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":6,"num_read":4158,"num_read_kb":177929,"num_write":699,"num_write_kb":267050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":272630673,"num_keys_recovered":47,"num_objects_omap":12,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":728,"ondisk_log_size":728,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2230208,"kb_used_data":418980,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564000832,"statfs":{"total":579820584960,"available":577536851968,"internally_reserved":0,"allocated":429035520,"data_stored":433022114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17825868,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":295,"num_read_kb":2238,"num_write":22,"num_write_kb":18440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":37748904,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019227"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":42,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.528482+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:50:52.528482+0000","last_peered":"2024-09-17T07:50:52.528482+0000","last_clean":"2024-09-17T07:50:52.528482+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:50:52.528482+0000","last_undegraded":"2024-09-17T07:50:52.528482+0000","last_fullsized":"2024-09-17T07:50:52.528482+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"127'8","reported_seq":44,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521535+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:50:51.521535+0000","last_peered":"2024-09-17T07:50:51.521535+0000","last_clean":"2024-09-17T07:50:51.521535+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:50:51.521535+0000","last_undegraded":"2024-09-17T07:50:51.521535+0000","last_fullsized":"2024-09-17T07:50:51.521535+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":7,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665207+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:50:50.665207+0000","last_peered":"2024-09-17T07:50:50.665207+0000","last_clean":"2024-09-17T07:50:50.665207+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:50:50.665207+0000","last_undegraded":"2024-09-17T07:50:50.665207+0000","last_fullsized":"2024-09-17T07:50:50.665207+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.183389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":820,"num_read_kb":28457,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":96,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665274+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:50:50.665274+0000","last_peered":"2024-09-17T07:50:50.665274+0000","last_clean":"2024-09-17T07:50:50.665274+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:50:50.665274+0000","last_undegraded":"2024-09-17T07:50:50.665274+0000","last_fullsized":"2024-09-17T07:50:50.665274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":112,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525822+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:50:52.525822+0000","last_peered":"2024-09-17T07:50:52.525822+0000","last_clean":"2024-09-17T07:50:52.525822+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:50:52.525822+0000","last_undegraded":"2024-09-17T07:50:52.525822+0000","last_fullsized":"2024-09-17T07:50:52.525822+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"127'270","reported_seq":1348,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.512071+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:50:50.512071+0000","last_peered":"2024-09-17T07:50:50.512071+0000","last_clean":"2024-09-17T07:50:50.512071+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:50:50.512071+0000","last_undegraded":"2024-09-17T07:50:50.512071+0000","last_fullsized":"2024-09-17T07:50:50.512071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39846109,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":1396,"num_read_kb":78369,"num_write":319,"num_write_kb":60555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":123,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665263+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:50:50.665263+0000","last_peered":"2024-09-17T07:50:50.665263+0000","last_clean":"2024-09-17T07:50:50.665263+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:50:50.665263+0000","last_undegraded":"2024-09-17T07:50:50.665263+0000","last_fullsized":"2024-09-17T07:50:50.665263+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"45'5","reported_seq":36,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T07:50:46.501809+0000","last_change":"2024-09-17T07:50:45.891597+0000","last_active":"2024-09-17T07:50:46.501809+0000","last_peered":"2024-09-17T07:50:46.501809+0000","last_clean":"2024-09-17T07:50:46.501809+0000","last_became_active":"2024-09-17T07:50:45.801562+0000","last_became_peered":"2024-09-17T07:50:45.801562+0000","last_unstale":"2024-09-17T07:50:46.501809+0000","last_undegraded":"2024-09-17T07:50:46.501809+0000","last_fullsized":"2024-09-17T07:50:46.501809+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:45.245789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'206","reported_seq":1105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.515384+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:50:51.515384+0000","last_peered":"2024-09-17T07:50:51.515384+0000","last_clean":"2024-09-17T07:50:51.515384+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:50:51.515384+0000","last_undegraded":"2024-09-17T07:50:51.515384+0000","last_fullsized":"2024-09-17T07:50:51.515384+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:00.953341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777455,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1556,"num_read_kb":46303,"num_write":103,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521586+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:50:51.521586+0000","last_peered":"2024-09-17T07:50:51.521586+0000","last_clean":"2024-09-17T07:50:51.521586+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:50:51.521586+0000","last_undegraded":"2024-09-17T07:50:51.521586+0000","last_fullsized":"2024-09-17T07:50:51.521586+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":212,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665226+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:50:50.665226+0000","last_peered":"2024-09-17T07:50:50.665226+0000","last_clean":"2024-09-17T07:50:50.665226+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:50:50.665226+0000","last_undegraded":"2024-09-17T07:50:50.665226+0000","last_fullsized":"2024-09-17T07:50:50.665226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":190,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665335+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:50:50.665335+0000","last_peered":"2024-09-17T07:50:50.665335+0000","last_clean":"2024-09-17T07:50:50.665335+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:50:50.665335+0000","last_undegraded":"2024-09-17T07:50:50.665335+0000","last_fullsized":"2024-09-17T07:50:50.665335+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"90'79","reported_seq":294,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.568181+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:50:50.568181+0000","last_peered":"2024-09-17T07:50:50.568181+0000","last_clean":"2024-09-17T07:50:50.568181+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:50:50.568181+0000","last_undegraded":"2024-09-17T07:50:50.568181+0000","last_fullsized":"2024-09-17T07:50:50.568181+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583080,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":89,"num_write_kb":24610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":54,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-17T07:50:46.055241+0000","last_change":"2024-09-17T07:50:46.054762+0000","last_active":"2024-09-17T07:50:46.055241+0000","last_peered":"2024-09-17T07:50:46.055241+0000","last_clean":"2024-09-17T07:50:46.055241+0000","last_became_active":"2024-09-17T07:50:45.803389+0000","last_became_peered":"2024-09-17T07:50:45.803389+0000","last_unstale":"2024-09-17T07:50:46.055241+0000","last_undegraded":"2024-09-17T07:50:46.055241+0000","last_fullsized":"2024-09-17T07:50:46.055241+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:46:23.290071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":205,"reported_epoch":129,"state":"peering","last_fresh":"2024-09-17T07:50:51.520412+0000","last_change":"2024-09-17T07:50:51.520678+0000","last_active":"2024-09-17T07:50:51.520377+0000","last_peered":"2024-09-17T07:50:50.665211+0000","last_clean":"2024-09-17T07:50:50.665211+0000","last_became_active":"2024-09-17T07:50:33.766392+0000","last_became_peered":"2024-09-17T07:50:33.766392+0000","last_unstale":"2024-09-17T07:50:51.520412+0000","last_undegraded":"2024-09-17T07:50:51.520412+0000","last_fullsized":"2024-09-17T07:50:51.520412+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:36.039448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":33554432,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":26,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.512039+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:50:50.512039+0000","last_peered":"2024-09-17T07:50:50.512039+0000","last_clean":"2024-09-17T07:50:50.512039+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:50:50.512039+0000","last_undegraded":"2024-09-17T07:50:50.512039+0000","last_fullsized":"2024-09-17T07:50:50.512039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665244+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:50:50.665244+0000","last_peered":"2024-09-17T07:50:50.665244+0000","last_clean":"2024-09-17T07:50:50.665244+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:50:50.665244+0000","last_undegraded":"2024-09-17T07:50:50.665244+0000","last_fullsized":"2024-09-17T07:50:50.665244+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":22,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521583+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:50:51.521583+0000","last_peered":"2024-09-17T07:50:51.521583+0000","last_clean":"2024-09-17T07:50:51.521583+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:50:51.521583+0000","last_undegraded":"2024-09-17T07:50:51.521583+0000","last_fullsized":"2024-09-17T07:50:51.521583+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":41,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.556404+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:50:52.556404+0000","last_peered":"2024-09-17T07:50:52.556404+0000","last_clean":"2024-09-17T07:50:52.556404+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:50:52.556404+0000","last_undegraded":"2024-09-17T07:50:52.556404+0000","last_fullsized":"2024-09-17T07:50:52.556404+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665342+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:50:50.665342+0000","last_peered":"2024-09-17T07:50:50.665342+0000","last_clean":"2024-09-17T07:50:50.665342+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:50:50.665342+0000","last_undegraded":"2024-09-17T07:50:50.665342+0000","last_fullsized":"2024-09-17T07:50:50.665342+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":35,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525665+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:50:52.525665+0000","last_peered":"2024-09-17T07:50:52.525665+0000","last_clean":"2024-09-17T07:50:52.525665+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:50:52.525665+0000","last_undegraded":"2024-09-17T07:50:52.525665+0000","last_fullsized":"2024-09-17T07:50:52.525665+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81789672,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3986,"num_read_kb":177795,"num_write":547,"num_write_kb":134331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":5,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166830080,"data_stored":176162256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217953,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":45,"num_write_kb":131085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":180355259,"num_keys_recovered":25,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251715584,"data_stored":251658690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860429,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369340,"kb_used_data":71208,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002500,"statfs":{"total":96636764160,"available":96258560000,"internally_reserved":0,"allocated":72916992,"data_stored":71985413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640307,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354300,"kb_used_data":48624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94017540,"statfs":{"total":96636764160,"available":96273960960,"internally_reserved":0,"allocated":49790976,"data_stored":55191646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.55900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54200000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39600000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.33500000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.27500000000000002}]}]},{"osd":3,"up_from":126,"seq":541165879298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328716,"kb_used_data":30780,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94043124,"statfs":{"total":96636764160,"available":96300158976,"internally_reserved":0,"allocated":31518720,"data_stored":30621693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360460,"kb_used_data":54784,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94011380,"statfs":{"total":96636764160,"available":96267653120,"internally_reserved":0,"allocated":56098816,"data_stored":55191905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.30299999999999999,"15min":0.32300000000000001},"min":{"1min":0.20300000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.441,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.29599999999999999,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.438,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.62,"5min":0.628,"15min":0.628},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.46400000000000002},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55200000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38100000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.373},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640308,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410192,"kb_used_data":104508,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93961648,"statfs":{"total":96636764160,"available":96216727552,"internally_reserved":0,"allocated":107016192,"data_stored":106113853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.32000000000000001,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.39000000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.33800000000000002,"15min":0.35599999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.45100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.32600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46500000000000002,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.504,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.51600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.51400000000000001,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53100000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48999999999999999}]}]},{"osd":0,"up_from":121,"seq":519691042819,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407200,"kb_used_data":109076,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93964640,"statfs":{"total":96636764160,"available":96219791360,"internally_reserved":0,"allocated":111693824,"data_stored":113917604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":67108997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16781312,"data_stored":16777254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43012096,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29380608,"data_stored":29360483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12583080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39866368,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:50:55.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T07:50:55.248 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.08089570800001, 2024-09-17T07:50:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:50:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:50:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:50:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.073236192, 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:50:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.06484662400001, 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:50:55.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:50:55.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:55.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.265 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.06447537400001, 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:50:55.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.05641240899999, 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:55.274 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.05604129700001, 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.04676089200001, 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:50:55.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.04510145899999, 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:50:55.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:50:55.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:50:55.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.039076634, 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.289 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.03882630699999, 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:50:55.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:50:55.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:50:55.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.03171246900001, 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:50:55.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:50:55.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:50:55.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.03119537000001, 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:55.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:50:55.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:50:55.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.01222420299999, 2024-09-17T07:50:55.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:50:55.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:50:55.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.00596734300001, 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:50:55.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:55.320 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 172.999431281, 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:50:55.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:50:55.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:50:55.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:50:55.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:50:55.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:50:55.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:50:55.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:50:55.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:50:55.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:50:55.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:50:55.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:50:55.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.626564871, 2024-09-17T07:50:55.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:50:55.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:55.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.62460446399999, 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.335 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 166.738935828, 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:50:55.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:50:55.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:55.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:55.347 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.42809019000001, 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:50:55.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.42672874499999, 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.359 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:55.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:55.361 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:55.360+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:55.461 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:55.461+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:55.504 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:55.504 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:55.505 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:55.512 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:55.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:50:55.562 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:50:55.562 INFO:teuthology.orchestra.run.smithi026.stdout:{"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":253,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":132,"num_osds":6,"num_up_osds":6,"osd_up_since":1726559448,"num_in_osds":6,"osd_in_since":1726559432,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"peering","count":1}],"num_pgs":21,"num_pools":4,"num_objects":101,"data_bytes":216598063,"bytes_used":2283732992,"bytes_avail":577536851968,"bytes_total":579820584960,"inactive_pgs_ratio":0.0476190485060215,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":9435652,"recovering_keys_per_sec":4,"num_objects_recovered":17,"num_bytes_recovered":37748904,"num_keys_recovered":17,"read_bytes_sec":572832,"write_bytes_sec":4719853,"read_op_per_sec":73,"write_op_per_sec":5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-17T07:47:53.419485+0000","services":{}},"progress_events":{}} 2024-09-17T07:50:55.562 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:55.562+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:55.575 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-17T07:50:55.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:50:55.660 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:55.660 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:55.661 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:55.661 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:55.661 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:55.662 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:55.661+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:55.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:55.762 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:55.762+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:55.775 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.51688188599999, 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:55.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:55.780 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 168.14857627000001, 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:55.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.456394015, 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:55.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.790 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.45053715899999, 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:55.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:55.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.43884420000001, 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.797 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:55.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.36998961699999, 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:55.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.358090328, 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:55.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.30724578100001, 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:55.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:55.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.820 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.301237574, 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:55.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.826 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.295537217, 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:55.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:55.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:55.832 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.28979759200001, 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:55.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.840 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.95668654400001, 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:55.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:55.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 160.111585647, 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:55.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:55.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 160.10549892399999, 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:55.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 160.022042623, 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 160.01572819099999, 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 160.009356731, 2024-09-17T07:50:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 160.003434255, 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:55.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.945451359, 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:55.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 155.987746687, 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:55.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:55.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:55.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:55.892 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:55.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:50:55.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:55.894 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:55.893+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:55.896 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:50:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:50:55.909 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-17T07:50:55.458848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216598063,"num_objects":101,"num_object_clones":2,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":6,"num_read":4158,"num_read_kb":177929,"num_write":699,"num_write_kb":267050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":272630673,"num_keys_recovered":47,"num_objects_omap":12,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":728,"ondisk_log_size":728,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2230208,"kb_used_data":418980,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564000832,"statfs":{"total":579820584960,"available":577536851968,"internally_reserved":0,"allocated":429035520,"data_stored":433022114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17825868,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":295,"num_read_kb":2238,"num_write":22,"num_write_kb":18440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":37748904,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000666"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":42,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.528482+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:50:52.528482+0000","last_peered":"2024-09-17T07:50:52.528482+0000","last_clean":"2024-09-17T07:50:52.528482+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:50:52.528482+0000","last_undegraded":"2024-09-17T07:50:52.528482+0000","last_fullsized":"2024-09-17T07:50:52.528482+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"127'8","reported_seq":44,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521535+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:50:51.521535+0000","last_peered":"2024-09-17T07:50:51.521535+0000","last_clean":"2024-09-17T07:50:51.521535+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:50:51.521535+0000","last_undegraded":"2024-09-17T07:50:51.521535+0000","last_fullsized":"2024-09-17T07:50:51.521535+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":7,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665207+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:50:50.665207+0000","last_peered":"2024-09-17T07:50:50.665207+0000","last_clean":"2024-09-17T07:50:50.665207+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:50:50.665207+0000","last_undegraded":"2024-09-17T07:50:50.665207+0000","last_fullsized":"2024-09-17T07:50:50.665207+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.183389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":820,"num_read_kb":28457,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":96,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665274+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:50:50.665274+0000","last_peered":"2024-09-17T07:50:50.665274+0000","last_clean":"2024-09-17T07:50:50.665274+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:50:50.665274+0000","last_undegraded":"2024-09-17T07:50:50.665274+0000","last_fullsized":"2024-09-17T07:50:50.665274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":112,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525822+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:50:52.525822+0000","last_peered":"2024-09-17T07:50:52.525822+0000","last_clean":"2024-09-17T07:50:52.525822+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:50:52.525822+0000","last_undegraded":"2024-09-17T07:50:52.525822+0000","last_fullsized":"2024-09-17T07:50:52.525822+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"127'270","reported_seq":1348,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.512071+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:50:50.512071+0000","last_peered":"2024-09-17T07:50:50.512071+0000","last_clean":"2024-09-17T07:50:50.512071+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:50:50.512071+0000","last_undegraded":"2024-09-17T07:50:50.512071+0000","last_fullsized":"2024-09-17T07:50:50.512071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39846109,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":1396,"num_read_kb":78369,"num_write":319,"num_write_kb":60555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":123,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665263+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:50:50.665263+0000","last_peered":"2024-09-17T07:50:50.665263+0000","last_clean":"2024-09-17T07:50:50.665263+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:50:50.665263+0000","last_undegraded":"2024-09-17T07:50:50.665263+0000","last_fullsized":"2024-09-17T07:50:50.665263+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"45'5","reported_seq":36,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T07:50:46.501809+0000","last_change":"2024-09-17T07:50:45.891597+0000","last_active":"2024-09-17T07:50:46.501809+0000","last_peered":"2024-09-17T07:50:46.501809+0000","last_clean":"2024-09-17T07:50:46.501809+0000","last_became_active":"2024-09-17T07:50:45.801562+0000","last_became_peered":"2024-09-17T07:50:45.801562+0000","last_unstale":"2024-09-17T07:50:46.501809+0000","last_undegraded":"2024-09-17T07:50:46.501809+0000","last_fullsized":"2024-09-17T07:50:46.501809+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:45.245789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'206","reported_seq":1105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.515384+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:50:51.515384+0000","last_peered":"2024-09-17T07:50:51.515384+0000","last_clean":"2024-09-17T07:50:51.515384+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:50:51.515384+0000","last_undegraded":"2024-09-17T07:50:51.515384+0000","last_fullsized":"2024-09-17T07:50:51.515384+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:00.953341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777455,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1556,"num_read_kb":46303,"num_write":103,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521586+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:50:51.521586+0000","last_peered":"2024-09-17T07:50:51.521586+0000","last_clean":"2024-09-17T07:50:51.521586+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:50:51.521586+0000","last_undegraded":"2024-09-17T07:50:51.521586+0000","last_fullsized":"2024-09-17T07:50:51.521586+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":212,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665226+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:50:50.665226+0000","last_peered":"2024-09-17T07:50:50.665226+0000","last_clean":"2024-09-17T07:50:50.665226+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:50:50.665226+0000","last_undegraded":"2024-09-17T07:50:50.665226+0000","last_fullsized":"2024-09-17T07:50:50.665226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":190,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665335+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:50:50.665335+0000","last_peered":"2024-09-17T07:50:50.665335+0000","last_clean":"2024-09-17T07:50:50.665335+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:50:50.665335+0000","last_undegraded":"2024-09-17T07:50:50.665335+0000","last_fullsized":"2024-09-17T07:50:50.665335+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"90'79","reported_seq":294,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.568181+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:50:50.568181+0000","last_peered":"2024-09-17T07:50:50.568181+0000","last_clean":"2024-09-17T07:50:50.568181+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:50:50.568181+0000","last_undegraded":"2024-09-17T07:50:50.568181+0000","last_fullsized":"2024-09-17T07:50:50.568181+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583080,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":89,"num_write_kb":24610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":54,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-17T07:50:46.055241+0000","last_change":"2024-09-17T07:50:46.054762+0000","last_active":"2024-09-17T07:50:46.055241+0000","last_peered":"2024-09-17T07:50:46.055241+0000","last_clean":"2024-09-17T07:50:46.055241+0000","last_became_active":"2024-09-17T07:50:45.803389+0000","last_became_peered":"2024-09-17T07:50:45.803389+0000","last_unstale":"2024-09-17T07:50:46.055241+0000","last_undegraded":"2024-09-17T07:50:46.055241+0000","last_fullsized":"2024-09-17T07:50:46.055241+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:46:23.290071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":205,"reported_epoch":129,"state":"peering","last_fresh":"2024-09-17T07:50:51.520412+0000","last_change":"2024-09-17T07:50:51.520678+0000","last_active":"2024-09-17T07:50:51.520377+0000","last_peered":"2024-09-17T07:50:50.665211+0000","last_clean":"2024-09-17T07:50:50.665211+0000","last_became_active":"2024-09-17T07:50:33.766392+0000","last_became_peered":"2024-09-17T07:50:33.766392+0000","last_unstale":"2024-09-17T07:50:51.520412+0000","last_undegraded":"2024-09-17T07:50:51.520412+0000","last_fullsized":"2024-09-17T07:50:51.520412+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:36.039448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":33554432,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":26,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.512039+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:50:50.512039+0000","last_peered":"2024-09-17T07:50:50.512039+0000","last_clean":"2024-09-17T07:50:50.512039+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:50:50.512039+0000","last_undegraded":"2024-09-17T07:50:50.512039+0000","last_fullsized":"2024-09-17T07:50:50.512039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665244+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:50:50.665244+0000","last_peered":"2024-09-17T07:50:50.665244+0000","last_clean":"2024-09-17T07:50:50.665244+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:50:50.665244+0000","last_undegraded":"2024-09-17T07:50:50.665244+0000","last_fullsized":"2024-09-17T07:50:50.665244+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":22,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521583+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:50:51.521583+0000","last_peered":"2024-09-17T07:50:51.521583+0000","last_clean":"2024-09-17T07:50:51.521583+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:50:51.521583+0000","last_undegraded":"2024-09-17T07:50:51.521583+0000","last_fullsized":"2024-09-17T07:50:51.521583+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":41,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.556404+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:50:52.556404+0000","last_peered":"2024-09-17T07:50:52.556404+0000","last_clean":"2024-09-17T07:50:52.556404+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:50:52.556404+0000","last_undegraded":"2024-09-17T07:50:52.556404+0000","last_fullsized":"2024-09-17T07:50:52.556404+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-17T07:50:50.665342+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:50:50.665342+0000","last_peered":"2024-09-17T07:50:50.665342+0000","last_clean":"2024-09-17T07:50:50.665342+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:50:50.665342+0000","last_undegraded":"2024-09-17T07:50:50.665342+0000","last_fullsized":"2024-09-17T07:50:50.665342+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":35,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525665+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:50:52.525665+0000","last_peered":"2024-09-17T07:50:52.525665+0000","last_clean":"2024-09-17T07:50:52.525665+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:50:52.525665+0000","last_undegraded":"2024-09-17T07:50:52.525665+0000","last_fullsized":"2024-09-17T07:50:52.525665+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81789672,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3986,"num_read_kb":177795,"num_write":547,"num_write_kb":134331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":5,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166830080,"data_stored":176162256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217953,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":45,"num_write_kb":131085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":180355259,"num_keys_recovered":25,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251715584,"data_stored":251658690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860429,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369340,"kb_used_data":71208,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002500,"statfs":{"total":96636764160,"available":96258560000,"internally_reserved":0,"allocated":72916992,"data_stored":71985413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640307,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354300,"kb_used_data":48624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94017540,"statfs":{"total":96636764160,"available":96273960960,"internally_reserved":0,"allocated":49790976,"data_stored":55191646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.55900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54200000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39600000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.33500000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.27500000000000002}]}]},{"osd":3,"up_from":126,"seq":541165879298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328716,"kb_used_data":30780,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94043124,"statfs":{"total":96636764160,"available":96300158976,"internally_reserved":0,"allocated":31518720,"data_stored":30621693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360460,"kb_used_data":54784,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94011380,"statfs":{"total":96636764160,"available":96267653120,"internally_reserved":0,"allocated":56098816,"data_stored":55191905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.30299999999999999,"15min":0.32300000000000001},"min":{"1min":0.20300000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.441,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.29599999999999999,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.438,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.62,"5min":0.628,"15min":0.628},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.46400000000000002},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55200000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38100000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.373},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640308,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410192,"kb_used_data":104508,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93961648,"statfs":{"total":96636764160,"available":96216727552,"internally_reserved":0,"allocated":107016192,"data_stored":106113853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.32000000000000001,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.39000000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.33800000000000002,"15min":0.35599999999999998},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.45100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.32600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46500000000000002,"15min":0.48299999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.504,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.51600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.51400000000000001,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53100000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48999999999999999}]}]},{"osd":0,"up_from":121,"seq":519691042819,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407200,"kb_used_data":109076,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93964640,"statfs":{"total":96636764160,"available":96219791360,"internally_reserved":0,"allocated":111693824,"data_stored":113917604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":67108997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16781312,"data_stored":16777254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43012096,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29380608,"data_stored":29360483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12583080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39866368,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:50:55.994 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:55.993+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:56.004 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:56.004 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:56.004 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:56.004 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:56.010 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:50:56.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:56.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:56.026 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:56.041 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:56.057 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:50:56.057 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:50:56.073 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:56.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:56.088 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:56.088 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:56.098 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:56.097+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:56.153 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:56.153 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:56.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:56.202 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:56.201+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:56.297 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:56.296+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:56.298 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:56.298 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:56.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:56.398 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:56.397+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:56.445 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:56.445 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:56.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:56.499 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:56.499+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:56.539 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:50:56.593 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:56.593 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:56.600 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:56.599+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:56.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:56.701 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:56.699+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:56.739 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:56.739 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:56.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:56.801 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:56.801+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:56.861 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:56.861 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:56.871 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:56.902 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:56.902+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:57.003 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:57.003+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:57.013 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.013 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:57.106 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:57.106+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:57.164 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.164 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:57.214 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:57.214+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:57.273 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.273 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:50:57.325 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:57.324+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:57.378 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.379 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:50:57.407 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:57.407+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:57.508 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:57.507+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:57.526 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.526 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:50:57.555 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-09-17T07:50:57.561 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:50:57.561 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:50:57.561 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:50:57.562 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:50:57.563 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:50:57.565 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:50:57.565 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:50:57.565 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-17T07:50:57.586 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:50:57.586 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:50:57.586 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100314 s, 105 MB/s 2024-09-17T07:50:57.587 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 29599 2024-09-17T07:50:57.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:50:57.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29599 2024-09-17T07:50:57.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:50:57.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:50:57.588 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:50:57.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:50:57.589 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:50:57.591 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:50:57.591 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:50:57.591 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:50:57.609 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:57.609+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:57.643 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.643 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:50:57.710 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:57.710+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:57.753 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.753 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:50:57.845 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:57.845+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:57.903 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:57.904 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:57.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:50:57.913 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:57.911+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:58.012 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:58.012+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:58.054 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:58.054 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:58.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:50:58.099 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:50:58.150 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:50:58.150+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:50:58.166 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:50:58.166 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:50:58.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:50:58.217 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:58.217+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:58.223 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:50:58.224 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:50:58.239 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:50:58.240 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:50:58.254 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:50:58.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:50:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:50:58.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:50:58.287 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:50:58.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:50:58.302 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:50:58.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:50:58.315 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:58.315+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:50:58.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.062482405, 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:50:58.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 170.69417678900001, 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.326 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:50:58.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.330 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 170.001994534, 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:50:58.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:50:58.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.336 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.996137678, 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:50:58.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:50:58.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.98444471900001, 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:50:58.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.347 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.91559013599999, 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:58.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:50:58.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:50:58.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:50:58.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.90369084700001, 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.353 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:50:58.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:50:58.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:50:58.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:58.358 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.85284630000001, 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:50:58.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:50:58.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.846838093, 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:50:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:50:58.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:50:58.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:50:58.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.84113773600001, 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.370 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:50:58.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:50:58.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.83539811099999, 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 165.50228706300001, 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.381 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.657186166, 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:50:58.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:50:58.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:50:58.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:50:58.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.65109944299999, 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.393 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.56764314200001, 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:50:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:50:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:50:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.56132871, 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.55495725, 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.409 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:50:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.54903477400001, 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:50:58.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:50:58.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 162.49105187800001, 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:50:58.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.533347206, 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:50:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:58.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:58.430 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:58.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:50:58.432 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:50:58.430+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:50:58.532 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:58.531+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:58.534 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:58.534 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:58.534 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:58.534 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:58.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:50:58.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:50:58.633 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:58.632+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:58.682 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:58.683 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:58.691 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:50:58.734 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:58.733+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.66637810899999, 2024-09-17T07:50:58.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:50:58.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.659038775, 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.841 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.65832170100001, 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:50:58.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.64909413999999, 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.846 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:50:58.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:50:58.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.63986957200001, 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.624394932, 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.855 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.615105422, 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:50:58.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:50:58.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.60851517699999, 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:50:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.592332696, 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:50:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.58325777600001, 2024-09-17T07:50:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.2428189, 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:50:58.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:50:58.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:50:58.882 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.24076662300001, 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:58.883 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:50:58.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:50:58.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.180534972, 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.889 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:50:58.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:50:58.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:58.895 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.01935582300001, 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:50:58.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.901 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.01933152199999, 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:50:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.01929714600001, 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:50:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.01927578900001, 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:50:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.01660714499999, 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:50:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:50:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.00688957200001, 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:50:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:50:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 159.04077918300001, 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:50:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:50:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:58.937 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:58.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:50:58.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:50:58.939 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:58.937+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:59.039 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:59.038+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:59.078 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:59.078 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:59.078 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:50:59.078 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:59.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:50:59.140 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:59.140+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:50:59.209 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:50:59.219 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-17T07:50:57.541657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216598063,"num_objects":101,"num_object_clones":2,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":6,"num_read":4158,"num_read_kb":177929,"num_write":699,"num_write_kb":267050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":301990820,"num_keys_recovered":50,"num_objects_omap":12,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":728,"ondisk_log_size":728,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2246792,"kb_used_data":435560,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":563984248,"statfs":{"total":579820584960,"available":577519869952,"internally_reserved":0,"allocated":446013440,"data_stored":449886576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":29360147,"num_keys_recovered":3,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.083145"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":42,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.528482+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:50:52.528482+0000","last_peered":"2024-09-17T07:50:52.528482+0000","last_clean":"2024-09-17T07:50:52.528482+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:50:52.528482+0000","last_undegraded":"2024-09-17T07:50:52.528482+0000","last_fullsized":"2024-09-17T07:50:52.528482+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"127'8","reported_seq":44,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521535+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:50:51.521535+0000","last_peered":"2024-09-17T07:50:51.521535+0000","last_clean":"2024-09-17T07:50:51.521535+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:50:51.521535+0000","last_undegraded":"2024-09-17T07:50:51.521535+0000","last_fullsized":"2024-09-17T07:50:51.521535+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":7,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"55'45","reported_seq":149,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546831+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:50:53.546831+0000","last_peered":"2024-09-17T07:50:53.546831+0000","last_clean":"2024-09-17T07:50:53.546831+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:50:53.546831+0000","last_undegraded":"2024-09-17T07:50:53.546831+0000","last_fullsized":"2024-09-17T07:50:53.546831+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.183389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583028,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":820,"num_read_kb":28457,"num_write":36,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":99,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546830+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:50:53.546830+0000","last_peered":"2024-09-17T07:50:53.546830+0000","last_clean":"2024-09-17T07:50:53.546830+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:50:53.546830+0000","last_undegraded":"2024-09-17T07:50:53.546830+0000","last_fullsized":"2024-09-17T07:50:53.546830+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":112,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525822+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:50:52.525822+0000","last_peered":"2024-09-17T07:50:52.525822+0000","last_clean":"2024-09-17T07:50:52.525822+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:50:52.525822+0000","last_undegraded":"2024-09-17T07:50:52.525822+0000","last_fullsized":"2024-09-17T07:50:52.525822+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"127'270","reported_seq":1351,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.528298+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:50:53.528298+0000","last_peered":"2024-09-17T07:50:53.528298+0000","last_clean":"2024-09-17T07:50:53.528298+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:50:53.528298+0000","last_undegraded":"2024-09-17T07:50:53.528298+0000","last_fullsized":"2024-09-17T07:50:53.528298+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39846109,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":1396,"num_read_kb":78369,"num_write":319,"num_write_kb":60555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":126,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546928+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:50:53.546928+0000","last_peered":"2024-09-17T07:50:53.546928+0000","last_clean":"2024-09-17T07:50:53.546928+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:50:53.546928+0000","last_undegraded":"2024-09-17T07:50:53.546928+0000","last_fullsized":"2024-09-17T07:50:53.546928+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"45'5","reported_seq":20,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.577034+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:50:53.577034+0000","last_peered":"2024-09-17T07:50:53.577034+0000","last_clean":"2024-09-17T07:50:53.577034+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:50:53.577034+0000","last_undegraded":"2024-09-17T07:50:53.577034+0000","last_fullsized":"2024-09-17T07:50:53.577034+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"127'206","reported_seq":1119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-17T07:50:56.680550+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:50:56.680550+0000","last_peered":"2024-09-17T07:50:56.680550+0000","last_clean":"2024-09-17T07:50:56.680550+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:50:56.680550+0000","last_undegraded":"2024-09-17T07:50:56.680550+0000","last_fullsized":"2024-09-17T07:50:56.680550+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:00.953341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777455,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1556,"num_read_kb":46303,"num_write":103,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521586+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:50:51.521586+0000","last_peered":"2024-09-17T07:50:51.521586+0000","last_clean":"2024-09-17T07:50:51.521586+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:50:51.521586+0000","last_undegraded":"2024-09-17T07:50:51.521586+0000","last_fullsized":"2024-09-17T07:50:51.521586+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":215,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546847+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:50:53.546847+0000","last_peered":"2024-09-17T07:50:53.546847+0000","last_clean":"2024-09-17T07:50:53.546847+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:50:53.546847+0000","last_undegraded":"2024-09-17T07:50:53.546847+0000","last_fullsized":"2024-09-17T07:50:53.546847+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":193,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546883+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:50:53.546883+0000","last_peered":"2024-09-17T07:50:53.546883+0000","last_clean":"2024-09-17T07:50:53.546883+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:50:53.546883+0000","last_undegraded":"2024-09-17T07:50:53.546883+0000","last_fullsized":"2024-09-17T07:50:53.546883+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"90'79","reported_seq":298,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-17T07:50:54.530925+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:50:54.530925+0000","last_peered":"2024-09-17T07:50:54.530925+0000","last_clean":"2024-09-17T07:50:54.530925+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:50:54.530925+0000","last_undegraded":"2024-09-17T07:50:54.530925+0000","last_fullsized":"2024-09-17T07:50:54.530925+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583080,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":214,"num_read_kb":24666,"num_write":89,"num_write_kb":24610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":14,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.578611+0000","last_change":"2024-09-17T07:50:53.578257+0000","last_active":"2024-09-17T07:50:53.578611+0000","last_peered":"2024-09-17T07:50:53.578611+0000","last_clean":"2024-09-17T07:50:53.578611+0000","last_became_active":"2024-09-17T07:50:53.530399+0000","last_became_peered":"2024-09-17T07:50:53.530399+0000","last_unstale":"2024-09-17T07:50:53.578611+0000","last_undegraded":"2024-09-17T07:50:53.578611+0000","last_fullsized":"2024-09-17T07:50:53.578611+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:09.511077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":229,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546845+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:50:53.546845+0000","last_peered":"2024-09-17T07:50:53.546845+0000","last_clean":"2024-09-17T07:50:53.546845+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:50:53.546845+0000","last_undegraded":"2024-09-17T07:50:53.546845+0000","last_fullsized":"2024-09-17T07:50:53.546845+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":29,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.528274+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:50:53.528274+0000","last_peered":"2024-09-17T07:50:53.528274+0000","last_clean":"2024-09-17T07:50:53.528274+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:50:53.528274+0000","last_undegraded":"2024-09-17T07:50:53.528274+0000","last_fullsized":"2024-09-17T07:50:53.528274+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":144,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546843+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:50:53.546843+0000","last_peered":"2024-09-17T07:50:53.546843+0000","last_clean":"2024-09-17T07:50:53.546843+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:50:53.546843+0000","last_undegraded":"2024-09-17T07:50:53.546843+0000","last_fullsized":"2024-09-17T07:50:53.546843+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":22,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-17T07:50:51.521583+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:50:51.521583+0000","last_peered":"2024-09-17T07:50:51.521583+0000","last_clean":"2024-09-17T07:50:51.521583+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:50:51.521583+0000","last_undegraded":"2024-09-17T07:50:51.521583+0000","last_fullsized":"2024-09-17T07:50:51.521583+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":41,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.556404+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:50:52.556404+0000","last_peered":"2024-09-17T07:50:52.556404+0000","last_clean":"2024-09-17T07:50:52.556404+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:50:52.556404+0000","last_undegraded":"2024-09-17T07:50:52.556404+0000","last_fullsized":"2024-09-17T07:50:52.556404+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":145,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-17T07:50:53.546899+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:50:53.546899+0000","last_peered":"2024-09-17T07:50:53.546899+0000","last_clean":"2024-09-17T07:50:53.546899+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:50:53.546899+0000","last_undegraded":"2024-09-17T07:50:53.546899+0000","last_fullsized":"2024-09-17T07:50:53.546899+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":35,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-17T07:50:52.525665+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:50:52.525665+0000","last_peered":"2024-09-17T07:50:52.525665+0000","last_clean":"2024-09-17T07:50:52.525665+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:50:52.525665+0000","last_undegraded":"2024-09-17T07:50:52.525665+0000","last_fullsized":"2024-09-17T07:50:52.525665+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81789672,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3986,"num_read_kb":177795,"num_write":547,"num_write_kb":134331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":5,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":41,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166830080,"data_stored":176162256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217953,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":71,"num_read_kb":50,"num_write":45,"num_write_kb":131085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":209715406,"num_keys_recovered":28,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268492800,"data_stored":268435906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860429,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369340,"kb_used_data":71208,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002500,"statfs":{"total":96636764160,"available":96258560000,"internally_reserved":0,"allocated":72916992,"data_stored":71985413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640308,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358456,"kb_used_data":52776,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94013384,"statfs":{"total":96636764160,"available":96269705216,"internally_reserved":0,"allocated":54042624,"data_stored":59407835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":1,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.55900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55900000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54200000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.33000000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.28999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.309},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.34699999999999998}]}]},{"osd":3,"up_from":126,"seq":541165879299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353336,"kb_used_data":55404,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94018504,"statfs":{"total":96636764160,"available":96274948096,"internally_reserved":0,"allocated":56733696,"data_stored":55809235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640308,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348220,"kb_used_data":42540,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94023620,"statfs":{"total":96636764160,"available":96280186880,"internally_reserved":0,"allocated":43560960,"data_stored":42630805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.29599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.432}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.35599999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46999999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640309,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410240,"kb_used_data":104556,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93961600,"statfs":{"total":96636764160,"available":96216678400,"internally_reserved":0,"allocated":107065344,"data_stored":106135684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.625}]},{"osd":5,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.67700000000000005}]}]},{"osd":0,"up_from":121,"seq":519691042819,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407200,"kb_used_data":109076,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93964640,"statfs":{"total":96636764160,"available":96219791360,"internally_reserved":0,"allocated":111693824,"data_stored":113917604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":67108997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12582931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43012096,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29380608,"data_stored":29360483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12583080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39866368,"data_stored":46137565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12583028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:50:59.221 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-17T07:50:59.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:50:59.221 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-17T07:50:59.221 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:50:59.221 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:50:59.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-09-17T07:50:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:59.226 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:50:59.226 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:50:59.226 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:50:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:59.235 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:50:59.276 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:50:59.275+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:50:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:59.322 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "" 2024-09-17T07:50:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:59.341 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:50:59.341+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:50:59.377 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:50:59.378 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:50:59.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:50:59.378 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:50:59.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.378 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:59.379 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.707771528000002, 2024-09-17T07:50:59.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:50:59.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.379 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.379 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:50:59.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:50:59.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:50:59.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:50:59.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:50:59.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.386 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:59.386 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.704758654999999, 2024-09-17T07:50:59.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:50:59.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:50:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.390 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.702491729999998, 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:50:59.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:50:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.700322112000002, 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:50:59.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.697032745000001, 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:50:59.399 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:50:59.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:50:59.403 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.691526615999997, 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:50:59.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.408 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.688688331000002, 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:50:59.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.684544492000001, 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:50:59.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.670309635999999, 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:59.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.420 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.659855180000001, 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:50:59.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:50:59.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:50:59.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:50:59.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.632145809999997, 2024-09-17T07:50:59.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.617605388000001, 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:50:59.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.613872252999997, 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:50:59.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.602830570000002, 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:50:59.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:50:59.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.588455783999997, 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.57449991, 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:59.448 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.570865827, 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:50:59.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:50:59.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.565859967, 2024-09-17T07:50:59.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.498579805, 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:50:59.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.489655737, 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:50:59.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:50:59.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:50:59.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:50:59.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:50:59.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.464 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:50:59.464 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:50:59.464 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:50:59.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:50:59.466 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:59.465+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:50:59.566 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:59.565+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:59.573 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:59.573 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:59.574 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:50:59.574 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:59.582 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:50:59.666 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:50:59.666+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:50:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:59.724 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:50:59.724 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:50:59.725 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:50:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:59.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:50:59.776 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:59.776+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:59.836 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.029409149999999, 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:50:59.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.027916126999999, 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:50:59.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:50:59.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.022930543999999, 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.018791021, 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.849 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:50:59.853 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.014006699999999, 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:50:59.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 14.006235453, 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:50:59.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.996197212, 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:50:59.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:50:59.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.976879652000001, 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:50:59.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.964481343999999, 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:50:59.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.960829788, 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.875 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:50:59.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.960246637000001, 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:50:59.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:50:59.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.955663295000001, 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:50:59.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.955334243999999, 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:59.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:50:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.947135048, 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:50:59.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 13.946613801, 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.896 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.316326994000001, 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:50:59.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.291977192999999, 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:59.905 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:50:59.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:50:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.290450384, 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:50:59.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.259205378000001, 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:50:59.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:50:59.918 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 10.207382319000001, 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:50:59.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:50:59.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:50:59.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:50:59.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:50:59.923 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:50:59.925 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:50:59.928 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:50:59.924+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:00.025 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:00.024+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:00.026 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:00.026 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:00.026 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:00.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:00.126 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:00.125+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:00.172 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:00.172 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:00.172 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:00.173 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:00.173 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:00.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:00.227 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:00.226+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:00.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:00.320 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:00.320 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.802832498000001, 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:00.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:00.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.787212832, 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.783132009999999, 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.328 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.776641868, 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:00.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.770926658, 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.763689023, 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:00.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.756158418, 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:00.343 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.752383183999999, 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.8010163779999999, 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.350 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:00.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:00.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7963075199999992, 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.354 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:00.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:00.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7924104679999999, 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7888212919999997, 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:00.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:00.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7843369750000004, 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:00.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7804049450000008, 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:00.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:00.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:00.374 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7766181060000008, 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:00.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:00.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7729664239999998, 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:00.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.382 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7688579830000002, 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:00.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7652579119999992, 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.387 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:00.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7618913190000001, 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:00.391 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:00.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:00.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 9.7584995849999991, 2024-09-17T07:51:00.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:00.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:00.399 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:00.400 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:00.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:00.402 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:00.400+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:00.436 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:00.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:00.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:00.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:00.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:00.482 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:00.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:00.502 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:00.501+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:00.506 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:00.506 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:00.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:00.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:00.608 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:00.608+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:00.666 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:00.666 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:00.715 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:00.715+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:00.804 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:00.804+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:00.816 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:00.816 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:00.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:00.905 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:00.904+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:00.926 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:00.926 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:00.935 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:01.019 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:01.019+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:01.077 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.077 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:01.127 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:01.126+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:01.207 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:01.207+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:01.228 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.228 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:01.309 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:01.308+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:01.345 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.345 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:01.409 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:01.409+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:01.453 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.453 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:01.510 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:01.510+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:01.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:51:01.562 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.562 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:01.611 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:01.611+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:01.709 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.710 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.712 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:01.711+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:01.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:01.813 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:01.813+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:01.816 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.816 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.824 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:01.914 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:01.913+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:01.961 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:01.961 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:01.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:02.015 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:02.014+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:02.072 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:02.072 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:02.081 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:02.121 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:02.121+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:02.216 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:02.216+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:02.221 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:02.222 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:02.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:02.317 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:02.317+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:02.332 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:02.332 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:02.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:02.418 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:02.418+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:02.439 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:02.439 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:02.448 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:02.519 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:02.519+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:02.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:51:02.586 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:02.587 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:02.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:02.638 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:02.638+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:02.648 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:02.664 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:02.664 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:02.678 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:02.678 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:02.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:02.694 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:02.695 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.177826190999999, 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:02.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:02.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.162206525, 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:02.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.158125703, 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:02.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:02.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.151635561000001, 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:02.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:02.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.145920351000001, 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:02.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.138682716, 2024-09-17T07:51:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.131152111, 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:02.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 13.127376877, 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:02.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.726 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.176010071, 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:02.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:02.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.171301213, 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:02.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:02.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.167404161, 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:02.735 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:02.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:02.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:02.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.163814985, 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:02.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.743 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.159330668000001, 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:02.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:02.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.155398637999999, 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:02.751 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.151611798999999, 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:02.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:02.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.147960117, 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:02.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:02.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.143851676000001, 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:02.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.140251605, 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:02.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.136885012, 2024-09-17T07:51:02.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:02.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:02.772 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 12.133493278, 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:02.776 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:02.777 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:02.778 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:02.779 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:02.777+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:02.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:02.780 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:02.879 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:02.878+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:02.921 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:02.921 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:02.930 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:02.980 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:02.979+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:03.031 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:03.032 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:03.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:03.080 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:03.080+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:03.181 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:03.181+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:03.183 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:03.183 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.01679734000001, 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.00913782399999, 2024-09-17T07:51:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:03.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:03.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.00074825600001, 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.00037700600001, 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:03.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:03.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.99231404099999, 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.991942929, 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.98266252400001, 2024-09-17T07:51:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.98100309099999, 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:03.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:03.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:03.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.97497826599999, 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:03.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.97472793899999, 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:03.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.96761410100001, 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:03.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:03.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:03.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.967097002, 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:03.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.94812583500001, 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.250 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.94186897500001, 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:03.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.935332913, 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:03.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:03.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.562466503, 2024-09-17T07:51:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.56050609600001, 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 174.67483745999999, 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:03.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 170.363991822, 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:03.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:03.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 170.36263037699999, 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.299 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:03.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:03.302 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:03.300+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:03.402 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:03.402+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:03.444 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:03.444 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:03.444 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:03.444 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:03.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:03.503 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:03.503+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:03.555 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:03.555 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:03.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:03.605 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:03.605+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:03.706 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:03.706+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:03.707 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:03.707 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.448910676, 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:03.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.712 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.08060506000001, 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:03.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.388422805, 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:03.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:03.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.382565949, 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.723 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.37087299000001, 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:03.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:03.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.30201840699999, 2024-09-17T07:51:03.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.29011911800001, 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.747 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.23927457100001, 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.233266364, 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.22756600700001, 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.22182638199999, 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 170.88871533400001, 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 168.043614437, 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 168.03752771399999, 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.95407141300001, 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.947756981, 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.941385521, 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:03.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.93546304500001, 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.810 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 167.87748014900001, 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 163.919775477, 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:03.826 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:03.828 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:03.828 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:03.827+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:03.928 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:03.927+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:03.932 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:03.932 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:03.933 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:03.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:04.029 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:04.029+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:04.039 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:04.039 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:04.039 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:04.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:04.130 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:04.129+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:04.146 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:04.146 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:04.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:04.146 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.97974477100001, 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.147 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:04.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:04.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.97240543699999, 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.152 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:04.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.156 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.971688363, 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:04.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.96246080200001, 2024-09-17T07:51:04.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:04.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.953236234, 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:04.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:04.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:04.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.93776159399999, 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:04.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:04.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.92847208399999, 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:04.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:04.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.92188183900001, 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:04.178 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.182 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.90569935799999, 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:04.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.186 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.896624438, 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:04.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.556185562, 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.191 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:04.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.55413328500001, 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:04.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:04.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:04.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:04.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.493901634, 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:04.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:04.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.33272248500001, 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:04.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:04.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:04.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:04.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:04.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.216 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.33269818400001, 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:04.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:04.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.33266380800001, 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:04.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:04.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:04.227 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.332642451, 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:04.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:04.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:04.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.32997380699999, 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:04.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:04.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.320256234, 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:04.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:04.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.243 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 164.35414584500001, 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.244 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:04.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:04.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:04.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.249 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:04.250 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:04.250+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:04.251 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:04.332 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:04.332 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:04.332 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-17T07:51:04.332 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:51:04.332 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:51:04.333 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-17T07:51:04.377 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:04.377+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:04.390 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:04.390 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:04.390 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:04.390 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:04.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:04.446 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:04.447 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:51:04.447 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:04.452 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:04.452+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:04.496 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:04.497 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:04.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:04.553 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:04.552+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:04.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:51:04.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:04.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29599\b' 2024-09-17T07:51:04.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29599\b' 2024-09-17T07:51:04.613 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:04.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:04.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29599\b' 2024-09-17T07:51:04.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29599\b' 2024-09-17T07:51:04.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:04.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:04.644 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.975469208, 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:04.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:04.649 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.972456334999997, 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:04.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.970189410000003, 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:04.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.968019792, 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:04.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.661 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.964730424999999, 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:04.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.959224296000002, 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.956386010999999, 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:04.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:04.674 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.952242171999998, 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:04.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.938007315999997, 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:04.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:04.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.927552859999999, 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:04.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.685 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.899843490000002, 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:04.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.885303067999999, 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:04.689 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:04.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.881569933000002, 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.87052825, 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.856153464000002, 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:04.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:04.704 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.842197589999998, 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:04.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.838563507, 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.708 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.833557646999999, 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:04.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.766277485, 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:04.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.757353417000001, 2024-09-17T07:51:04.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:04.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:04.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:04.723 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:04.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:04.725 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:04.724+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:04.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:04.825 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:04.825+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:04.829 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:04.829 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:04.829 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:04.829 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:04.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:04.900 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:04.900 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:04.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:04.927 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:04.926+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:04.931 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:04.946 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:04.963 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:04.979 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:04.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:04.981 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:04.981 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:04.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:05.028 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:05.028+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:05.122 INFO:tasks.workunit.client.0.smithi098.stderr:+ LOG_FILE=/tmp/tmp.TyfTMF4aW5/rbd-nbd.log 2024-09-17T07:51:05.122 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -n '' ']' 2024-09-17T07:51:05.122 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd27356 --log-file=/tmp/tmp.TyfTMF4aW5/rbd-nbd.log 2024-09-17T07:51:05.122 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:05.122 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:05.123 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:05.123 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:05.125 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:05.126 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:05.126 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd27356 --log-file=/tmp/tmp.TyfTMF4aW5/rbd-nbd.log 2024-09-17T07:51:05.129 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:05.129+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:05.132 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.132 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:05.226 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd0 2024-09-17T07:51:05.226 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:05.226 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:05.226 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:05.226 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:05.226 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:05.229 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:05.229+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:05.245 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.245 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:05.281 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29796 2024-09-17T07:51:05.281 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29796 2024-09-17T07:51:05.281 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29796 -C rbd-nbd 2024-09-17T07:51:05.286 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:05.286 INFO:tasks.workunit.client.0.smithi098.stdout: 29796 ? 00:00:00 rbd-nbd 2024-09-17T07:51:05.287 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo mkfs /dev/nbd0 2024-09-17T07:51:05.287 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:05.287 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:05.287 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:05.288 INFO:tasks.workunit.client.0.smithi098.stderr:++ which mkfs 2024-09-17T07:51:05.290 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/mkfs 2024-09-17T07:51:05.290 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:05.290 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd0 2024-09-17T07:51:05.331 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:05.331+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:05.390 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.390 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:05.442 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:05.442+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:05.495 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.496 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:05.545 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:05.545+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:05.558 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:51:05.606 INFO:tasks.workunit.client.0.smithi098.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T07:51:05.612 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.612 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:05.633 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:05.633+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout:Discarding device blocks: 0/65536 done 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout:Creating filesystem with 65536 1k blocks and 16384 inodes 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout:Filesystem UUID: 119a2138-c608-4906-a3bb-1989a1eaa02a 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout:Superblock backups stored on blocks: 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout: 8193, 24577, 40961, 57345 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-17T07:51:05.642 INFO:tasks.workunit.client.0.smithi098.stdout:Allocating group tables: 0/8 done 2024-09-17T07:51:05.659 INFO:tasks.workunit.client.0.smithi098.stdout:Writing inode tables: 0/8 done 2024-09-17T07:51:05.724 INFO:tasks.workunit.client.0.smithi098.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2024-09-17T07:51:05.725 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-17T07:51:05.726 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:05.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:05.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:05.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:05.728 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:05.728 INFO:tasks.workunit.client.0.smithi098.stderr:++ which mount 2024-09-17T07:51:05.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/mount 2024-09-17T07:51:05.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:05.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:05.733 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:05.733+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:05.739 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.740 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:05.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create rbd/testrbdnbd27356@quiesce3 2024-09-17T07:51:05.835 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:05.834+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:05.892 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:05.892 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:05.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:05.936 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:05.935+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:06.036 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:06.036+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:06.042 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.043 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:06.137 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:06.137+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:06.152 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.152 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:06.244 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:06.244+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:06.302 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.303 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:06.356 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:06.355+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:06.410 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.410 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.419 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:06.440 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:06.440+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:06.541 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:06.541+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:06.558 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.558 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:06.570 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-17T07:51:06.575 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/tmp/tmp.TyfTMF4aW5/mnt/test bs=1M count=1 oflag=direct 2024-09-17T07:51:06.576 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:06.576 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:06.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:06.577 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:51:06.579 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:51:06.579 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:06.579 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/tmp/tmp.TyfTMF4aW5/mnt/test bs=1M count=1 oflag=direct 2024-09-17T07:51:06.592 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:51:06.592 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:51:06.592 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241867 s, 434 MB/s 2024-09-17T07:51:06.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo umount /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:06.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:06.594 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:06.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:06.595 INFO:tasks.workunit.client.0.smithi098.stderr:++ which umount 2024-09-17T07:51:06.597 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/umount 2024-09-17T07:51:06.597 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:06.597 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:06.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 29796 2024-09-17T07:51:06.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:51:06.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=29796 2024-09-17T07:51:06.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:51:06.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:06.621 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:06.622 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:06.622 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:06.623 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:06.624 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:06.624 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:51:06.642 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:06.642+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:06.665 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.666 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:06.691 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:06.691 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^29796\b' 2024-09-17T07:51:06.691 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^29796\b' 2024-09-17T07:51:06.729 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:06.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:06.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *29796\b' 2024-09-17T07:51:06.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *29796\b' 2024-09-17T07:51:06.735 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:06.736 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:06.758 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:06.758+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:06.812 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.813 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:06.862 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:06.862+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:06.945 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:06.944+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:06.959 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:06.959 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:06.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:06.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:07.045 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:07.045+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:07.105 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:07.105 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:07.114 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:07.117 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:07.133 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:07.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:07.146 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:07.146+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:07.148 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:07.148 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:07.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:07.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:07.179 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:07.180 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:07.195 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:07.195 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:07.217 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:07.217 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:07.217 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:07.218 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:07.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat /tmp/tmp.TyfTMF4aW5/rbd-nbd.log 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stdout:2024-09-17T07:51:05.193+0000 7f51e6446800 0 ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable), process rbd-nbd, pid 29787 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stdout:2024-09-17T07:51:05.194+0000 7f51e6446800 0 pidfile_write: ignore empty --pid-file 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stdout:2024-09-17T07:51:05.611+0000 7f51e3b4d640 0 -- 172.21.15.98:0/3362086757 >> [v2:172.21.15.94:6808/31605,v1:172.21.15.94:6809/31605] conn(0x55a46b487810 msgr2=0x55a46b489bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stdout:2024-09-17T07:51:05.625+0000 7f51e3b4d640 0 -- 172.21.15.98:0/3362086757 >> [v2:172.21.15.94:6808/31605,v1:172.21.15.94:6809/31605] conn(0x7f51d003e6e0 msgr2=0x7f51d0040ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:51:07.237 INFO:tasks.workunit.client.0.smithi098.stdout:2024-09-17T07:51:06.680+0000 7f51c3fff640 0 rbd-nbd: disconnect request received 2024-09-17T07:51:07.238 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.TyfTMF4aW5/rbd-nbd.log 2024-09-17T07:51:07.238 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'quiesce failed' /tmp/tmp.TyfTMF4aW5/rbd-nbd.log 2024-09-17T07:51:07.238 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:07.239 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd27356 2024-09-17T07:51:07.239 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:07.239 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:07.240 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.240 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:07.242 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:07.242 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:07.242 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd27356 2024-09-17T07:51:07.247 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:07.247+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:07.341 INFO:tasks.workunit.client.0.smithi098.stderr:+ OUT='/dev/nbd0 0051fd3c-4652-4a19-86ff-ca89f87b0625' 2024-09-17T07:51:07.342 INFO:tasks.workunit.client.0.smithi098.stderr:+ read DEV COOKIE 2024-09-17T07:51:07.342 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:07.342 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:07.342 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:07.342 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:07.342 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:07.348 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:07.348+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:07.371 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:07.371 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:07.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:07.394 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=29924 2024-09-17T07:51:07.394 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 29924 2024-09-17T07:51:07.394 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 29924 -C rbd-nbd 2024-09-17T07:51:07.399 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:07.399 INFO:tasks.workunit.client.0.smithi098.stdout: 29924 ? 00:00:00 rbd-nbd 2024-09-17T07:51:07.400 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:07.400 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:07.400 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:07.400 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.400 INFO:tasks.workunit.client.0.smithi098.stderr:++ which mount 2024-09-17T07:51:07.402 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/mount 2024-09-17T07:51:07.402 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:07.402 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:07.421 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device detach rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.421 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:07.421 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:07.422 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.422 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:07.423 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:07.424 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:07.424 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.463 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:07.463+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:07.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false get_pid rbd 2024-09-17T07:51:07.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:07.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:07.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:07.493 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:07.493 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:07.524 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.717306319999999, 2024-09-17T07:51:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:07.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:07.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:07.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:07.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:07.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:07.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:07.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.715813297, 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:07.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.710827714000001, 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.706688191000001, 2024-09-17T07:51:07.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.701903869999999, 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:07.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:07.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:07.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.694132623000002, 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.684094382000001, 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.554 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:07.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.664776822, 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:07.558 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:07.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:07.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:07.562 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.652378513999999, 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.566 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.648726958000001, 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:07.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.648143807, 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.643560465, 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:07.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.643231413999999, 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.579 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:07.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.635032217999999, 2024-09-17T07:51:07.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:07.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:07.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.634510971000001, 2024-09-17T07:51:07.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:07.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 18.004224164, 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:07.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:07.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.979874363, 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:07.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:07.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:07.600 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.978347553999999, 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:07.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:07.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.947102548, 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:07.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 17.895279489, 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:07.609 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:07.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:07.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:07.614 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:07.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:51:07.616 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:07.615+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:07.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID= 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n '' 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 1 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:07.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: could not validate attach request 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: must specify --cookie or --force to proceed 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -n 0051fd3c-4652-4a19-86ff-ca89f87b0625 ']' 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie 0051fd3c-4652-4a19-86ff-ca89f87b0625 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:07.620 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:07.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:07.621 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie 0051fd3c-4652-4a19-86ff-ca89f87b0625 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.685 INFO:tasks.workunit.client.0.smithi098.stdout:/dev/nbd0 2024-09-17T07:51:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:07.692 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:07.692 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:07.716 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:07.715+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:07.720 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:07.720 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:07.720 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:07.720 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:07.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:07.748 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30002 2024-09-17T07:51:07.748 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30002 2024-09-17T07:51:07.748 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30002 -C rbd-nbd 2024-09-17T07:51:07.753 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:07.753 INFO:tasks.workunit.client.0.smithi098.stdout: 30002 ? 00:00:00 rbd-nbd 2024-09-17T07:51:07.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-17T07:51:07.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:07.753 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:07.754 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.754 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:07.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:07.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:07.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-17T07:51:07.817 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:07.816+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:07.827 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false get_pid rbd 2024-09-17T07:51:07.828 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:07.828 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:07.828 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:07.828 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:07.828 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:07.867 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:07.867 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:07.867 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:07.867 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:07.868 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:07.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID= 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n '' 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 1 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -n 0051fd3c-4652-4a19-86ff-ca89f87b0625 ']' 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie 0051fd3c-4652-4a19-86ff-ca89f87b0625 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:07.881 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:07.882 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:07.882 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:07.883 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:07.883 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:07.884 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie 0051fd3c-4652-4a19-86ff-ca89f87b0625 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:07.918 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:07.917+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:07.976 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:07.976 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:07.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:07.976 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.458268048000001, 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:07.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.442648382000002, 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:07.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:07.984 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.438567559999999, 2024-09-17T07:51:07.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:07.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:07.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:07.985 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:07.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:07.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:07.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:07.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:07.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:07.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.432077417999999, 2024-09-17T07:51:07.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:07.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:07.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:07.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.426362208, 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:07.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:07.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.419124573000001, 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:08.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.003 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.411593967999998, 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:08.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.407818733999999, 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.456451928, 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.451743069999999, 2024-09-17T07:51:08.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:08.017 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.017 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.017 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:08.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:08.021 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.447846018, 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.444256842000001, 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:08.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.439772524999999, 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:08.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.435840495000001, 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.034 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:08.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.432053656000001, 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:08.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.428401974, 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:08.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.424293533, 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:08.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.420693461999999, 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:08.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.417326869, 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:08.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:08.058 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 17.413935134999999, 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:08.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:08.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:08.063 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:08.063 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:08.064 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:08.064+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:08.064 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:08.164 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:08.164+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:08.166 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:08.167 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:08.167 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:08.167 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:08.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:08.184 INFO:tasks.workunit.client.0.smithi098.stdout:/dev/nbd0 2024-09-17T07:51:08.189 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:08.189 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:08.189 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:08.190 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:08.190 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:08.247 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30064 2024-09-17T07:51:08.247 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30064 2024-09-17T07:51:08.247 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30064 -C rbd-nbd 2024-09-17T07:51:08.252 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:08.252 INFO:tasks.workunit.client.0.smithi098.stdout: 30064 ? 00:00:00 rbd-nbd 2024-09-17T07:51:08.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ ls /tmp/tmp.TyfTMF4aW5/mnt/ 2024-09-17T07:51:08.254 INFO:tasks.workunit.client.0.smithi098.stdout:lost+found 2024-09-17T07:51:08.254 INFO:tasks.workunit.client.0.smithi098.stdout:test 2024-09-17T07:51:08.255 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/tmp/tmp.TyfTMF4aW5/mnt/test of=/dev/null bs=1M count=1 2024-09-17T07:51:08.262 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:51:08.262 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:51:08.262 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622776 s, 168 MB/s 2024-09-17T07:51:08.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo dd if=/tmp/tmp.TyfTMF4aW5/data of=/tmp/tmp.TyfTMF4aW5/mnt/test1 bs=1M count=1 oflag=direct 2024-09-17T07:51:08.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:08.262 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:08.263 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:08.263 INFO:tasks.workunit.client.0.smithi098.stderr:++ which dd 2024-09-17T07:51:08.265 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/dd 2024-09-17T07:51:08.265 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:08.265 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.TyfTMF4aW5/data of=/tmp/tmp.TyfTMF4aW5/mnt/test1 bs=1M count=1 oflag=direct 2024-09-17T07:51:08.265 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:08.265+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:08.288 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-17T07:51:08.288 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-17T07:51:08.288 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141436 s, 74.1 MB/s 2024-09-17T07:51:08.289 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo umount /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:08.289 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:08.289 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:08.290 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:08.291 INFO:tasks.workunit.client.0.smithi098.stderr:++ which umount 2024-09-17T07:51:08.292 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/umount 2024-09-17T07:51:08.292 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:08.293 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:08.318 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 30064 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=30064 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:08.319 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:08.320 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:08.320 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:08.320 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:08.320 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:08.320 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:08.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:08.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:08.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:51:08.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:08.366 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:08.366+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:08.467 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:08.467+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.30656240100001, 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.29890288499999, 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.29051331700001, 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.290142067, 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:08.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.28207910200001, 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:08.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:08.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.28170799, 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.499 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:08.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:08.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.272427585, 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:08.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:08.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:08.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.27076815199999, 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.509 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:08.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:08.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.26474332699999, 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.514 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:08.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:08.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.519 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.26449299999999, 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:08.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:08.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.25737916200001, 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:08.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:08.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:08.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.256862063, 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:08.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:08.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.536 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.23789089600001, 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:08.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:08.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.231634036, 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.22509797399999, 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:08.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:08.551 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.85223156399999, 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:08.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:08.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.85027115700001, 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:08.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:08.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 179.96460252099999, 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:08.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:08.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.653756883, 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.573 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:08.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 175.65239543800001, 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:08.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:08.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.586 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:08.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:51:08.589 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:08.587+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:08.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:08.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^30064\b' 2024-09-17T07:51:08.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^30064\b' 2024-09-17T07:51:08.633 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:08.633 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:08.633 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *30064\b' 2024-09-17T07:51:08.634 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *30064\b' 2024-09-17T07:51:08.639 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:08.639 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:08.689 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:08.689+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:08.730 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:08.730 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:08.730 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:08.731 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:08.739 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:08.789 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:08.789+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:08.840 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:08.841 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:08.841 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:08.841 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:08.841 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:08.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:08.891 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:08.891+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:08.992 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:08.992+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:08.993 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:08.993 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.734943668, 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.36663805200001, 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:08.998 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:08.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.67445579700001, 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.668598941, 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.65690598200001, 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.58805139899999, 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.57615211000001, 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.52530756300001, 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.519299356, 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.51359899900001, 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.50785937399999, 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.17474832600001, 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.329647429, 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:09.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.32356070599999, 2024-09-17T07:51:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:09.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.24010440500001, 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.233789973, 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.227418513, 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.22149603700001, 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 173.16351314100001, 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 169.205808469, 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:09.107 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:09.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:09.109 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:09.108+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:09.140 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -n 0051fd3c-4652-4a19-86ff-ca89f87b0625 ']' 2024-09-17T07:51:09.140 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd --show-cookie --cookie 'abc de' --options try-netlink map rbd/testrbdnbd27356 2024-09-17T07:51:09.140 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:09.140 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:09.141 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:09.141 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:09.143 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:09.143 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:09.143 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie --cookie 'abc de' --options try-netlink map rbd/testrbdnbd27356 2024-09-17T07:51:09.207 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:09.210 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:09.209+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:09.215 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:09.215 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:09.215 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:09.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:09.251 INFO:tasks.workunit.client.0.smithi098.stderr:+ OUT='/dev/nbd0 abc de' 2024-09-17T07:51:09.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ read DEV ANOTHER_COOKIE 2024-09-17T07:51:09.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:09.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:09.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:09.252 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:09.252 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30146 2024-09-17T07:51:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30146 2024-09-17T07:51:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30146 -C rbd-nbd 2024-09-17T07:51:09.309 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:09.309+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:09.311 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:09.311 INFO:tasks.workunit.client.0.smithi098.stdout: 30146 ? 00:00:00 rbd-nbd 2024-09-17T07:51:09.311 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 'abc de' = 'abc de' 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd0 30146 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd0 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=30146 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:09.313 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:09.314 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:09.314 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:09.314 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-17T07:51:09.325 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:09.326 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:09.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:09.327 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:09.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:09.342 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:09.343 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:09.357 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:09.358 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:09.373 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:09.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:09.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^30146\b' 2024-09-17T07:51:09.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^30146\b' 2024-09-17T07:51:09.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:09.405 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:09.405 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:09.409 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:09.409+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:09.421 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:09.422 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:09.422 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *30146\b' 2024-09-17T07:51:09.422 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *30146\b' 2024-09-17T07:51:09.427 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:09.427 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:09.429 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:09.429 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:09.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:09.457 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:09.457 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:09.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:09.511 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:09.511+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:09.535 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:09.535 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:09.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:09.611 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:09.611+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:09.681 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:09.682 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:09.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:09.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:09.732 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:09.733 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:09.733+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:09.744 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-17T07:51:09.461149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":228691655,"num_objects":122,"num_object_clones":19,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":13,"num_read":4962,"num_read_kb":181725,"num_write":878,"num_write_kb":273697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":301990820,"num_keys_recovered":50,"num_objects_omap":12,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":43,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":889,"ondisk_log_size":889,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2257248,"kb_used_data":446032,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":563973792,"statfs":{"total":579820584960,"available":577509163008,"internally_reserved":0,"allocated":456736768,"data_stored":461117349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11044940,"num_objects":18,"num_object_clones":16,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":7,"num_read":660,"num_read_kb":2678,"num_write":171,"num_write_kb":5617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001044"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.617379+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:51:05.617379+0000","last_peered":"2024-09-17T07:51:05.617379+0000","last_clean":"2024-09-17T07:51:05.617379+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:51:05.617379+0000","last_undegraded":"2024-09-17T07:51:05.617379+0000","last_fullsized":"2024-09-17T07:51:05.617379+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"134'9","reported_seq":55,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.611426+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:51:05.611426+0000","last_peered":"2024-09-17T07:51:05.611426+0000","last_clean":"2024-09-17T07:51:05.611426+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:05.611426+0000","last_undegraded":"2024-09-17T07:51:05.611426+0000","last_fullsized":"2024-09-17T07:51:05.611426+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777292,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":8,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"138'51","reported_seq":254,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.571662+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:51:06.571662+0000","last_peered":"2024-09-17T07:51:06.571662+0000","last_clean":"2024-09-17T07:51:06.571662+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:51:06.571662+0000","last_undegraded":"2024-09-17T07:51:06.571662+0000","last_fullsized":"2024-09-17T07:51:06.571662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:24.024613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777370,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":916,"num_read_kb":29888,"num_write":39,"num_write_kb":16450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":104,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558651+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:51:05.558651+0000","last_peered":"2024-09-17T07:51:05.558651+0000","last_clean":"2024-09-17T07:51:05.558651+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:51:05.558651+0000","last_undegraded":"2024-09-17T07:51:05.558651+0000","last_fullsized":"2024-09-17T07:51:05.558651+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":119,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.617310+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:51:05.617310+0000","last_peered":"2024-09-17T07:51:05.617310+0000","last_clean":"2024-09-17T07:51:05.617310+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:51:05.617310+0000","last_undegraded":"2024-09-17T07:51:05.617310+0000","last_fullsized":"2024-09-17T07:51:05.617310+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"138'378","reported_seq":1612,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.612549+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:51:06.612549+0000","last_peered":"2024-09-17T07:51:06.612549+0000","last_clean":"2024-09-17T07:51:06.612549+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:06.612549+0000","last_undegraded":"2024-09-17T07:51:06.612549+0000","last_fullsized":"2024-09-17T07:51:06.612549+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45636829,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":1822,"num_read_kb":80492,"num_write":462,"num_write_kb":65576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":131,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558751+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:51:05.558751+0000","last_peered":"2024-09-17T07:51:05.558751+0000","last_clean":"2024-09-17T07:51:05.558751+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:51:05.558751+0000","last_undegraded":"2024-09-17T07:51:05.558751+0000","last_fullsized":"2024-09-17T07:51:05.558751+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"138'6","reported_seq":28,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.565755+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:51:06.565755+0000","last_peered":"2024-09-17T07:51:06.565755+0000","last_clean":"2024-09-17T07:51:06.565755+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:06.565755+0000","last_undegraded":"2024-09-17T07:51:06.565755+0000","last_fullsized":"2024-09-17T07:51:06.565755+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582969,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":6,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"138'239","reported_seq":1235,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.685143+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:51:06.685143+0000","last_peered":"2024-09-17T07:51:06.685143+0000","last_clean":"2024-09-17T07:51:06.685143+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:06.685143+0000","last_undegraded":"2024-09-17T07:51:06.685143+0000","last_fullsized":"2024-09-17T07:51:06.685143+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:11:35.472036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18095381,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":1833,"num_read_kb":46536,"num_write":125,"num_write_kb":33820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.611530+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:51:05.611530+0000","last_peered":"2024-09-17T07:51:05.611530+0000","last_clean":"2024-09-17T07:51:05.611530+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:51:05.611530+0000","last_undegraded":"2024-09-17T07:51:05.611530+0000","last_fullsized":"2024-09-17T07:51:05.611530+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":220,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558597+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:51:05.558597+0000","last_peered":"2024-09-17T07:51:05.558597+0000","last_clean":"2024-09-17T07:51:05.558597+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:51:05.558597+0000","last_undegraded":"2024-09-17T07:51:05.558597+0000","last_fullsized":"2024-09-17T07:51:05.558597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":198,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558530+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:05.558530+0000","last_peered":"2024-09-17T07:51:05.558530+0000","last_clean":"2024-09-17T07:51:05.558530+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:05.558530+0000","last_undegraded":"2024-09-17T07:51:05.558530+0000","last_fullsized":"2024-09-17T07:51:05.558530+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"137'91","reported_seq":315,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.559464+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:51:06.559464+0000","last_peered":"2024-09-17T07:51:06.559464+0000","last_clean":"2024-09-17T07:51:06.559464+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:06.559464+0000","last_undegraded":"2024-09-17T07:51:06.559464+0000","last_fullsized":"2024-09-17T07:51:06.559464+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373608,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":217,"num_read_kb":24675,"num_write":98,"num_write_kb":25130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":19,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.610186+0000","last_change":"2024-09-17T07:50:53.578257+0000","last_active":"2024-09-17T07:51:05.610186+0000","last_peered":"2024-09-17T07:51:05.610186+0000","last_clean":"2024-09-17T07:51:05.610186+0000","last_became_active":"2024-09-17T07:50:53.530399+0000","last_became_peered":"2024-09-17T07:50:53.530399+0000","last_unstale":"2024-09-17T07:51:05.610186+0000","last_undegraded":"2024-09-17T07:51:05.610186+0000","last_fullsized":"2024-09-17T07:51:05.610186+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:09.511077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"45'6","reported_seq":234,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558735+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:51:05.558735+0000","last_peered":"2024-09-17T07:51:05.558735+0000","last_clean":"2024-09-17T07:51:05.558735+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:05.558735+0000","last_undegraded":"2024-09-17T07:51:05.558735+0000","last_fullsized":"2024-09-17T07:51:05.558735+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":34,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.610162+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:51:05.610162+0000","last_peered":"2024-09-17T07:51:05.610162+0000","last_clean":"2024-09-17T07:51:05.610162+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:51:05.610162+0000","last_undegraded":"2024-09-17T07:51:05.610162+0000","last_fullsized":"2024-09-17T07:51:05.610162+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":149,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558665+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:51:05.558665+0000","last_peered":"2024-09-17T07:51:05.558665+0000","last_clean":"2024-09-17T07:51:05.558665+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:05.558665+0000","last_undegraded":"2024-09-17T07:51:05.558665+0000","last_fullsized":"2024-09-17T07:51:05.558665+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":30,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.611446+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:51:05.611446+0000","last_peered":"2024-09-17T07:51:05.611446+0000","last_clean":"2024-09-17T07:51:05.611446+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:51:05.611446+0000","last_undegraded":"2024-09-17T07:51:05.611446+0000","last_fullsized":"2024-09-17T07:51:05.611446+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":49,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.611587+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:51:05.611587+0000","last_peered":"2024-09-17T07:51:05.611587+0000","last_clean":"2024-09-17T07:51:05.611587+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:05.611587+0000","last_undegraded":"2024-09-17T07:51:05.611587+0000","last_fullsized":"2024-09-17T07:51:05.611587+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.558555+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:51:05.558555+0000","last_peered":"2024-09-17T07:51:05.558555+0000","last_clean":"2024-09-17T07:51:05.558555+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:51:05.558555+0000","last_undegraded":"2024-09-17T07:51:05.558555+0000","last_fullsized":"2024-09-17T07:51:05.558555+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":42,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-17T07:51:05.617473+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:51:05.617473+0000","last_peered":"2024-09-17T07:51:05.617473+0000","last_clean":"2024-09-17T07:51:05.617473+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:51:05.617473+0000","last_undegraded":"2024-09-17T07:51:05.617473+0000","last_fullsized":"2024-09-17T07:51:05.617473+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":93883188,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":13,"num_read":4788,"num_read_kb":181591,"num_write":724,"num_write_kb":140976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":5,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":43,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178188288,"data_stored":189051496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218029,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":73,"num_read_kb":50,"num_write":47,"num_write_kb":131087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":209715406,"num_keys_recovered":28,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268509184,"data_stored":268436058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":49,"ondisk_log_size":49,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860432,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369488,"kb_used_data":71364,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002352,"statfs":{"total":96636764160,"available":96258408448,"internally_reserved":0,"allocated":73076736,"data_stored":72091274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.42399999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.30299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640310,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361368,"kb_used_data":55696,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010472,"statfs":{"total":96636764160,"available":96266723328,"internally_reserved":0,"allocated":57032704,"data_stored":62362162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40200000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.36599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.29899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.28499999999999998}]}]},{"osd":3,"up_from":126,"seq":541165879301,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353924,"kb_used_data":55988,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94017916,"statfs":{"total":96636764160,"available":96274345984,"internally_reserved":0,"allocated":57331712,"data_stored":56370198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640310,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349840,"kb_used_data":44160,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94022000,"statfs":{"total":96636764160,"available":96278528000,"internally_reserved":0,"allocated":45219840,"data_stored":44250898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.222}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.254},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.23499999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34100000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.33000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.376}]}]},{"osd":1,"up_from":11,"seq":47244640311,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411400,"kb_used_data":105720,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93960440,"statfs":{"total":96636764160,"available":96215490560,"internally_reserved":0,"allocated":108257280,"data_stored":107288833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.254},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.34699999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.379},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.40899999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.45600000000000002}]}]},{"osd":0,"up_from":121,"seq":519691042822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411228,"kb_used_data":113104,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93960612,"statfs":{"total":96636764160,"available":96215666688,"internally_reserved":0,"allocated":115818496,"data_stored":118753984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47022080,"data_stored":50923741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30515200,"data_stored":30485935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":30952893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13127680,"data_stored":13115560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43876352,"data_stored":50923741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:51:09.787 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:09.788 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:09.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:09.840 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:09.840+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:09.898 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:09.899 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:09.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:09.915 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:09.914+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:09.928 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:51:09.929 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd snap ls rbd/testrbdnbd27356 2024-09-17T07:51:09.929 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-17T07:51:09.969 INFO:tasks.workunit.client.0.smithi098.stderr:+ SNAPID=4 2024-09-17T07:51:09.969 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd27356 2024-09-17T07:51:09.969 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:09.969 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:09.970 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:09.970 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:09.972 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:09.972 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:09.972 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd27356 2024-09-17T07:51:10.015 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:10.015+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:10.050 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.050 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:10.081 INFO:tasks.workunit.client.0.smithi098.stderr:+ OUT='/dev/nbd0 3ad82d63-af5c-4e0a-9b9c-38b5afeb3581' 2024-09-17T07:51:10.081 INFO:tasks.workunit.client.0.smithi098.stderr:+ read DEV COOKIE 2024-09-17T07:51:10.081 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:10.081 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:10.081 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:10.081 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:10.082 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:10.116 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:10.115+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:10.137 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30234 2024-09-17T07:51:10.137 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30234 2024-09-17T07:51:10.137 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30234 -C rbd-nbd 2024-09-17T07:51:10.142 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:10.142 INFO:tasks.workunit.client.0.smithi098.stdout: 30234 ? 00:00:00 rbd-nbd 2024-09-17T07:51:10.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device detach rbd/testrbdnbd27356 --snap-id 4 --device-type nbd 2024-09-17T07:51:10.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:10.143 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:10.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:10.144 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:10.145 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:10.145 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:10.146 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd27356 --snap-id 4 --device-type nbd 2024-09-17T07:51:10.163 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.163 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:10.217 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:10.217+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:10.222 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false get_pid rbd 2024-09-17T07:51:10.222 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:10.222 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:10.222 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:10.222 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:10.222 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:10.272 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.273 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID= 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n '' 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 1 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:10.275 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:10.276 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:10.276 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:10.276 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:10.277 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:10.278 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:10.278 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:10.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:10.310 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: could not validate attach request 2024-09-17T07:51:10.311 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-17T07:51:10.311 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: must specify --cookie or --force to proceed 2024-09-17T07:51:10.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:10.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -n 3ad82d63-af5c-4e0a-9b9c-38b5afeb3581 ']' 2024-09-17T07:51:10.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie 3ad82d63-af5c-4e0a-9b9c-38b5afeb3581 --snap-id 4 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:10.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:10.313 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:10.314 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:10.314 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:10.316 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:10.316 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:10.316 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie 3ad82d63-af5c-4e0a-9b9c-38b5afeb3581 --snap-id 4 rbd/testrbdnbd27356 --device-type nbd 2024-09-17T07:51:10.325 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:10.325+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:10.379 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.379 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:10.425 INFO:tasks.workunit.client.0.smithi098.stdout:/dev/nbd0 2024-09-17T07:51:10.430 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:10.430 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:10.430 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:10.430 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:10.430 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:10.431 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:10.430+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:10.489 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30304 2024-09-17T07:51:10.490 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30304 2024-09-17T07:51:10.490 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30304 -C rbd-nbd 2024-09-17T07:51:10.494 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:10.495 INFO:tasks.workunit.client.0.smithi098.stdout: 30304 ? 00:00:00 rbd-nbd 2024-09-17T07:51:10.495 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-17T07:51:10.495 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:10.495 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:10.496 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:10.496 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:10.498 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:10.498 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:10.498 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-17T07:51:10.519 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:10.519+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:10.527 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.527 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:10.574 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false get_pid rbd 2024-09-17T07:51:10.574 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:10.575 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:10.575 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:10.575 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:10.575 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd0'\'']/id' 2024-09-17T07:51:10.592 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:51:10.627 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID= 2024-09-17T07:51:10.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n '' 2024-09-17T07:51:10.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 1 2024-09-17T07:51:10.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:10.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:51:10.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd config image set rbd/testrbdnbd27356 rbd_discard_granularity_bytes 4096 2024-09-17T07:51:10.636 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:10.636+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:10.640 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.640 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:10.683 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd feature enable rbd/testrbdnbd27356 journaling 2024-09-17T07:51:10.721 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:10.720+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:10.743 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:51:10.743 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:10.743 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:10.744 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:10.744 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:10.746 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:10.746 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:10.746 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:51:10.796 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.796 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:10.848 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:10.848+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:10.907 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:10.908 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:10.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:10.957 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:10.957+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:11.022 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:11.022+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:11.058 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:11.059 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:11.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:11.151 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:11.151+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:11.209 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:11.209 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:11.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:11.258 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:11.258+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:11.325 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:11.325+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:11.359 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:11.359 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:11.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:11.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:11.426 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:11.426+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:11.474 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:11.475 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:11.484 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:11.527 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:11.526+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:11.532 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:11.548 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:11.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:11.562 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:11.563 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:11.575 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:51:11.580 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:11.596 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:11.612 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:11.612 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:11.624 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:11.624 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:11.624 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:11.624 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:11.624 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:11.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:11.665 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:11.664+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:11.728 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:11.728+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.732 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.56550458199999, 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:11.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:11.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.737 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.55816524799999, 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:11.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:11.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.557448174, 2024-09-17T07:51:11.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:11.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.54822061300001, 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.747 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.538996045, 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.523521405, 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:11.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.51423189499999, 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:11.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:11.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.50764165000001, 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:11.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.491459169, 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:11.774 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.48238424900001, 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:11.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:11.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 184.141945373, 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:11.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:11.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 184.13989309600001, 2024-09-17T07:51:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 184.079661445, 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.795 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:11.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:11.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:11.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.91848229600001, 2024-09-17T07:51:11.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:11.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.806 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.91845799500001, 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:11.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:11.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:11.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.91842361900001, 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:11.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:11.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:11.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.918402262, 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:11.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:11.822 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.91573361799999, 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:11.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:11.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.827 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.906016045, 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:11.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:11.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:11.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 171.93990565600001, 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:11.833 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:11.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:11.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:11.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.839 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:11.839 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:11.839 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:11.840 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:11.839+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:11.841 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:11.939 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:11.940 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:11.940 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:11.940 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:11.941 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:11.941+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:11.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:12.042 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:12.041+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:12.085 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:12.085 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:12.085 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:12.086 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:12.086 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:12.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:12.142 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:12.142+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:12.192 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:12.192 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:12.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:12.192 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:12.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.192 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.193 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:12.193 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.522866547000007, 2024-09-17T07:51:12.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:12.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.519853674000004, 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:12.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:12.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.517586749000003, 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:12.204 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:12.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.208 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.515417131000007, 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:12.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:12.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.512127763999999, 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.506621635000002, 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.503783350000006, 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.499639510999998, 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:12.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.485404654999996, 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:12.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:12.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.474950199000006, 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.447240828999995, 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.432700406999999, 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:12.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.428967271999994, 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:12.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:12.249 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.417925589000006, 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:12.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.403550803000002, 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:12.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.389594928999998, 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:12.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:12.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:12.261 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.385960846, 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.380954985999999, 2024-09-17T07:51:12.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.313674824, 2024-09-17T07:51:12.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 26.304750756000001, 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:12.273 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:12.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:12.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:12.277 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:12.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:12.279 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:12.279+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:12.380 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:12.379+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:12.386 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:12.386 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:12.387 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:12.387 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:12.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:12.480 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:12.480+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:12.496 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:12.496 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:12.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:12.581 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:12.581+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.841419987999998, 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.839926965, 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.834941382, 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:12.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:12.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:12.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:12.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:12.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.830801859000001, 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:12.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:12.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.826017537999999, 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:12.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:12.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.818246291000001, 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:12.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.808208050000001, 2024-09-17T07:51:12.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.679 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.78889049, 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.776492181999998, 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.772840626000001, 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.772257475, 2024-09-17T07:51:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.767674133, 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.767345081999999, 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.759145885999999, 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 26.758624639000001, 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.128337832, 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.103988031, 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.102461221999999, 2024-09-17T07:51:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.071216216, 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.019393157, 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:12.736 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:12.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:12.738 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:12.736+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:12.838 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:12.838 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:12.838 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:12.838 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:12.838 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:12.838+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:12.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:12.939 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:12.939+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:12.943 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:12.943 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:12.943 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:12.943 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:12.943 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:12.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:13.040 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:13.040+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:13.091 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:13.092 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.573899646000001, 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:13.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.558279980000002, 2024-09-17T07:51:13.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.554199157999999, 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:13.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:13.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.547709015999999, 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:13.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:13.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.541993806000001, 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.113 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.534756171000002, 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:13.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.527225565999998, 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:13.117 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:13.120 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 23.523450331999999, 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.572083526, 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:13.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.567374667999999, 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.563477616, 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:13.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:13.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:13.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.559888440000002, 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:13.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.555404122999999, 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.551472093000001, 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.145 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:13.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.547685254000001, 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:13.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:13.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.544033572, 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:13.153 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:13.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.539925131, 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:13.157 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:13.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.536325059999999, 2024-09-17T07:51:13.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.532958467, 2024-09-17T07:51:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:13.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.529566732999999, 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:13.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:13.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:13.173 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:13.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:13.175 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:13.175+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:13.275 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:13.275+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:13.281 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:13.281 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:13.282 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:13.282 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:13.290 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:13.376 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:13.376+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:13.433 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:13.433 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:13.433 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:13.433 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:13.433 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:13.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:13.485 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:13.485+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:13.545 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.37862966399999, 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.546 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:13.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.370970148, 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:13.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:13.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:13.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.36258058000001, 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.556 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:13.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:13.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.36220933000001, 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.562 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:13.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.566 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.35414636499999, 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:13.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:13.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.35377525300001, 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.571 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:13.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:13.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:13.576 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.34449484800001, 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:13.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.342835415, 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:13.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:13.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.33681059, 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.336560263, 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.32944642499999, 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.32892932600001, 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:13.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.30995815899999, 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:13.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:13.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.30370129900001, 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:13.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:13.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.297165237, 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:13.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:13.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:13.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:13.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.924298827, 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:13.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:13.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.92233841999999, 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:13.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:13.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.036669784, 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:13.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.72582414600001, 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:13.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.72446270099999, 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:13.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:13.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:13.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:13.651 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:13.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:51:13.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:13.654 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:13.653+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:13.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:13.782 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:13.782+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:13.815 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:13.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:13.832 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:13.845 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:13.846 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:13.847 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:13.854 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:13.854+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:13.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:13.864 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:13.864 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:13.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:13.880 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:13.896 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:13.954 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:13.954+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:13.966 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:13.967 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:13.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:14.055 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:14.055+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:14.117 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.118 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:14.171 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:14.171+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:14.236 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.236 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:14.256 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:14.257+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:14.357 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:14.356+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:14.384 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.384 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:14.458 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:14.458+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:14.490 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.490 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:14.559 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:14.559+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:51:14.599 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.599 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:14.660 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:14.660+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:14.705 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.705 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:14.747 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:14.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:14.747 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:51:14.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:51:14.791 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:14.791+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:14.811 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.811 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:14.863 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:14.863+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:14.925 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:14.926 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:14.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:14.962 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:14.962+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:15.057 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 141, 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:51:14.572995+0000", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:50:48.503993+0000", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:50:32.754467+0000", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-09-17T07:51:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 7, 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:51:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "138", 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 8, 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 138, 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:51:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:51:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:51:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:51:15.076 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "138", 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 8, 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 138, 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:51:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "new_default_pool27356", 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:48:02.539579+0000", 2024-09-17T07:51:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:51:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "49", 2024-09-17T07:51:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 49, 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 38, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 121, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/69763", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/69763", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/69763", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/69763", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:51:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 127, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 49, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 126, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/61678", 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/61678", 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/61678", 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/61678", 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:51:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:51:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 68, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/57427", 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/57427", 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/57427", 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/57427", 2024-09-17T07:51:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:51:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-09-17T07:51:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "4.0", 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-09-17T07:51:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:51:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:51:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:51:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:15.133 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:15.135 INFO:tasks.thrashosds.thrasher:j is {'epoch': 141, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:51:14.572995+0000', 'last_up_change': '2024-09-17T07:50:48.503993+0000', 'last_in_change': '2024-09-17T07:50:32.754467+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 38, 'up_from': 121, 'up_thru': 129, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 69763}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 69763}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 69763}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 69763}]}, 'public_addr': '172.21.15.26:6801/69763', 'cluster_addr': '172.21.15.26:6803/69763', 'heartbeat_back_addr': '172.21.15.26:6807/69763', 'heartbeat_front_addr': '172.21.15.26:6805/69763', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 11, 'last_clean_end': 49, 'up_from': 126, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 61678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 61678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 61678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 61678}]}, 'public_addr': '172.21.15.94:6817/61678', 'cluster_addr': '172.21.15.94:6819/61678', 'heartbeat_back_addr': '172.21.15.94:6823/61678', 'heartbeat_front_addr': '172.21.15.94:6821/61678', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 68, 'up_from': 96, 'up_thru': 129, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 57427}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 57427}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 57427}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 57427}]}, 'public_addr': '172.21.15.94:6801/57427', 'cluster_addr': '172.21.15.94:6803/57427', 'heartbeat_back_addr': '172.21.15.94:6807/57427', 'heartbeat_front_addr': '172.21.15.94:6805/57427', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}, {'pgid': '3.1', 'osds': [3, 2]}, {'pgid': '4.0', 'osds': [4, 5]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:51:15.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:15.136 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.136 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.137 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:15.137 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:15.134+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:15.234 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:15.234+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:15.280 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.281 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:15.335 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:15.335+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:15.391 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.391 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:15.441 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:15.441+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:15.450 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:15.462 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-17T07:51:14.577784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229742279,"num_objects":124,"num_object_clones":19,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":12,"num_read":5917,"num_read_kb":188110,"num_write":928,"num_write_kb":274751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":301990820,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":950,"ondisk_log_size":950,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2260500,"kb_used_data":449296,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":563970540,"statfs":{"total":579820584960,"available":577505832960,"internally_reserved":0,"allocated":460079104,"data_stored":465142914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1050624,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":-1,"num_read":955,"num_read_kb":6385,"num_write":50,"num_write_kb":1054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.116910"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.747939+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:51:08.747939+0000","last_peered":"2024-09-17T07:51:08.747939+0000","last_clean":"2024-09-17T07:51:08.747939+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:51:08.747939+0000","last_undegraded":"2024-09-17T07:51:08.747939+0000","last_fullsized":"2024-09-17T07:51:08.747939+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"134'9","reported_seq":57,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.563619+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:51:08.563619+0000","last_peered":"2024-09-17T07:51:08.563619+0000","last_clean":"2024-09-17T07:51:08.563619+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:08.563619+0000","last_undegraded":"2024-09-17T07:51:08.563619+0000","last_fullsized":"2024-09-17T07:51:08.563619+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777292,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":8,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"138'52","reported_seq":423,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:10.821278+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:51:10.821278+0000","last_peered":"2024-09-17T07:51:10.821278+0000","last_clean":"2024-09-17T07:51:10.821278+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:51:10.821278+0000","last_undegraded":"2024-09-17T07:51:10.821278+0000","last_fullsized":"2024-09-17T07:51:10.821278+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:24.024613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16778394,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":1082,"num_read_kb":32099,"num_write":40,"num_write_kb":16451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.587311+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:51:08.587311+0000","last_peered":"2024-09-17T07:51:08.587311+0000","last_clean":"2024-09-17T07:51:08.587311+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:51:08.587311+0000","last_undegraded":"2024-09-17T07:51:08.587311+0000","last_fullsized":"2024-09-17T07:51:08.587311+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":121,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.747904+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:51:08.747904+0000","last_peered":"2024-09-17T07:51:08.747904+0000","last_clean":"2024-09-17T07:51:08.747904+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:51:08.747904+0000","last_undegraded":"2024-09-17T07:51:08.747904+0000","last_fullsized":"2024-09-17T07:51:08.747904+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"139'405","reported_seq":1799,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:10.813603+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:51:10.813603+0000","last_peered":"2024-09-17T07:51:10.813603+0000","last_clean":"2024-09-17T07:51:10.813603+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:10.813603+0000","last_undegraded":"2024-09-17T07:51:10.813603+0000","last_fullsized":"2024-09-17T07:51:10.813603+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686429,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":2009,"num_read_kb":84177,"num_write":490,"num_write_kb":66616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":133,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.587344+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:51:08.587344+0000","last_peered":"2024-09-17T07:51:08.587344+0000","last_clean":"2024-09-17T07:51:08.587344+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:51:08.587344+0000","last_undegraded":"2024-09-17T07:51:08.587344+0000","last_fullsized":"2024-09-17T07:51:08.587344+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"138'6","reported_seq":29,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.565927+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:51:06.565927+0000","last_peered":"2024-09-17T07:51:06.565927+0000","last_clean":"2024-09-17T07:51:06.565927+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:06.565927+0000","last_undegraded":"2024-09-17T07:51:06.565927+0000","last_fullsized":"2024-09-17T07:51:06.565927+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582969,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":6,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"139'271","reported_seq":1403,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:10.821647+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:51:10.821647+0000","last_peered":"2024-09-17T07:51:10.821647+0000","last_clean":"2024-09-17T07:51:10.821647+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:10.821647+0000","last_undegraded":"2024-09-17T07:51:10.821647+0000","last_fullsized":"2024-09-17T07:51:10.821647+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:13.532835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18095381,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":2428,"num_read_kb":47021,"num_write":139,"num_write_kb":33826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.563704+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:51:08.563704+0000","last_peered":"2024-09-17T07:51:08.563704+0000","last_clean":"2024-09-17T07:51:08.563704+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:51:08.563704+0000","last_undegraded":"2024-09-17T07:51:08.563704+0000","last_fullsized":"2024-09-17T07:51:08.563704+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":222,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.587287+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:51:08.587287+0000","last_peered":"2024-09-17T07:51:08.587287+0000","last_clean":"2024-09-17T07:51:08.587287+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:51:08.587287+0000","last_undegraded":"2024-09-17T07:51:08.587287+0000","last_fullsized":"2024-09-17T07:51:08.587287+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.587611+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:08.587611+0000","last_peered":"2024-09-17T07:51:08.587611+0000","last_clean":"2024-09-17T07:51:08.587611+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:08.587611+0000","last_undegraded":"2024-09-17T07:51:08.587611+0000","last_fullsized":"2024-09-17T07:51:08.587611+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"137'91","reported_seq":320,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:10.423348+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:51:10.423348+0000","last_peered":"2024-09-17T07:51:10.423348+0000","last_clean":"2024-09-17T07:51:10.423348+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:10.423348+0000","last_undegraded":"2024-09-17T07:51:10.423348+0000","last_fullsized":"2024-09-17T07:51:10.423348+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373608,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":222,"num_read_kb":24678,"num_write":98,"num_write_kb":25130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":20,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.563307+0000","last_change":"2024-09-17T07:50:53.578257+0000","last_active":"2024-09-17T07:51:06.563307+0000","last_peered":"2024-09-17T07:51:06.563307+0000","last_clean":"2024-09-17T07:51:06.563307+0000","last_became_active":"2024-09-17T07:50:53.530399+0000","last_became_peered":"2024-09-17T07:50:53.530399+0000","last_unstale":"2024-09-17T07:51:06.563307+0000","last_undegraded":"2024-09-17T07:51:06.563307+0000","last_fullsized":"2024-09-17T07:51:06.563307+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:09.511077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"139'7","reported_seq":239,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:10.727190+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:51:10.727190+0000","last_peered":"2024-09-17T07:51:10.727190+0000","last_clean":"2024-09-17T07:51:10.727190+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:10.727190+0000","last_undegraded":"2024-09-17T07:51:10.727190+0000","last_fullsized":"2024-09-17T07:51:10.727190+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":13,"num_write_kb":16393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T07:51:06.563306+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:51:06.563306+0000","last_peered":"2024-09-17T07:51:06.563306+0000","last_clean":"2024-09-17T07:51:06.563306+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:51:06.563306+0000","last_undegraded":"2024-09-17T07:51:06.563306+0000","last_fullsized":"2024-09-17T07:51:06.563306+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.587332+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:51:08.587332+0000","last_peered":"2024-09-17T07:51:08.587332+0000","last_clean":"2024-09-17T07:51:08.587332+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:08.587332+0000","last_undegraded":"2024-09-17T07:51:08.587332+0000","last_fullsized":"2024-09-17T07:51:08.587332+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":32,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.563647+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:51:08.563647+0000","last_peered":"2024-09-17T07:51:08.563647+0000","last_clean":"2024-09-17T07:51:08.563647+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:51:08.563647+0000","last_undegraded":"2024-09-17T07:51:08.563647+0000","last_fullsized":"2024-09-17T07:51:08.563647+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.563730+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:51:08.563730+0000","last_peered":"2024-09-17T07:51:08.563730+0000","last_clean":"2024-09-17T07:51:08.563730+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:08.563730+0000","last_undegraded":"2024-09-17T07:51:08.563730+0000","last_fullsized":"2024-09-17T07:51:08.563730+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":152,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.587355+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:51:08.587355+0000","last_peered":"2024-09-17T07:51:08.587355+0000","last_clean":"2024-09-17T07:51:08.587355+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:51:08.587355+0000","last_undegraded":"2024-09-17T07:51:08.587355+0000","last_fullsized":"2024-09-17T07:51:08.587355+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":44,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T07:51:08.748034+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:51:08.748034+0000","last_peered":"2024-09-17T07:51:08.748034+0000","last_clean":"2024-09-17T07:51:08.748034+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:51:08.748034+0000","last_undegraded":"2024-09-17T07:51:08.748034+0000","last_fullsized":"2024-09-17T07:51:08.748034+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":94933812,"num_objects":67,"num_object_clones":19,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":5741,"num_read_kb":187975,"num_write":767,"num_write_kb":142023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":6,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180305920,"data_stored":191155816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218029,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":75,"num_read_kb":51,"num_write":54,"num_write_kb":131094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":209715406,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268509184,"data_stored":268436058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860433,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369512,"kb_used_data":71388,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002328,"statfs":{"total":96636764160,"available":96258383872,"internally_reserved":0,"allocated":73101312,"data_stored":72102084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.29199999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640311,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363488,"kb_used_data":57812,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94008352,"statfs":{"total":96636764160,"available":96264552448,"internally_reserved":0,"allocated":59199488,"data_stored":65284952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.625}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.60099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.46200000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.439},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.47999999999999998}]}]},{"osd":3,"up_from":126,"seq":541165879302,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353944,"kb_used_data":56012,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94017896,"statfs":{"total":96636764160,"available":96274325504,"internally_reserved":0,"allocated":57356288,"data_stored":56381008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349860,"kb_used_data":44184,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94021980,"statfs":{"total":96636764160,"available":96278507520,"internally_reserved":0,"allocated":45244416,"data_stored":44262418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.29799999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.502}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.49199999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.65400000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640312,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411424,"kb_used_data":105752,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93960416,"statfs":{"total":96636764160,"available":96215465984,"internally_reserved":0,"allocated":108290048,"data_stored":107301415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.215}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.255},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.26900000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.311}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.371}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.32000000000000001}]}]},{"osd":0,"up_from":121,"seq":519691042823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412272,"kb_used_data":114148,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93959568,"statfs":{"total":96636764160,"available":96214597632,"internally_reserved":0,"allocated":116887552,"data_stored":119811037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48078848,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30519296,"data_stored":30486959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":30952893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13127680,"data_stored":13115560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44933120,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:51:15.464 INFO:tasks.thrashosds.thrasher:Setting 1.7 to [2, 5, 4, 3] 2024-09-17T07:51:15.464 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.7', '2', '5', '4', '3'] 2024-09-17T07:51:15.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 1.7 2 5 4 3 2024-09-17T07:51:15.537 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:15.536+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:15.537 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.537 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:15.637 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:15.637+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:15.643 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.643 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:15.739 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:15.738+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:15.754 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.754 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:15.840 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:15.839+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:15.865 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.865 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:15.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:15.941 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:15.940+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:15.977 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:15.977 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:15.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:16.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:16.036 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:16.041 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:16.040+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:16.051 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:16.051 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:16.066 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:16.066 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:16.083 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:16.083 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:16.090 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:16.090 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:16.090 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:16.099 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:16.099 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:16.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:16.114 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:16.114 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:16.142 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:16.141+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:16.215 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:16.216 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:16.216 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:16.216 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:16.216 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:16.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:16.267 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:16.267+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:16.328 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 194.06857665699999, 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:16.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:16.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:16.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 188.70027104100001, 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 188.008088786, 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.337 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:16.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:16.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.342 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 188.00223192999999, 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:16.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.99053897100001, 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:16.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:16.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:16.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.92168438799999, 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:16.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:16.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.909785099, 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:16.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.85894055200001, 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.366 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:16.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.852932345, 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.372 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:16.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.847231988, 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.378 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:16.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:16.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.84149236299999, 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.384 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:16.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:16.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.50838131500001, 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:16.390 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:16.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:16.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.663280418, 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.396 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:16.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.65719369499999, 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.402 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:16.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.407 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.57373739400001, 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.408 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:16.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:16.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:16.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.413 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.56742296199999, 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:16.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.561051502, 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:16.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:16.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:16.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.555129026, 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:16.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 180.49714613, 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.539441458, 2024-09-17T07:51:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:16.441 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:16.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:16.444 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:16.443+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:16.543 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:16.543+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:16.552 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:16.552 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:16.553 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:16.553 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:16.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.7 pg_upmap_items mapping to [2->5,4->3] 2024-09-17T07:51:16.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:51:16.644 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:16.643+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:16.660 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:16.660 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:16.660 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:16.669 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:16.745 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:16.745+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:16.768 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:16.768 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.602089605, 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.769 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:16.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.59475027100001, 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.59403319699999, 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.584805636, 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:16.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.57558106799999, 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.786 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.790 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.56010642800001, 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.55081691800001, 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:16.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:16.799 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.544226673, 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.52804419200001, 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.804 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:16.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.808 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.51896927199999, 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:16.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.17853039600001, 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:16.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.176478119, 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:16.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:16.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.11624646800001, 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:16.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:16.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.95506731899999, 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:16.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.955043018, 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:16.835 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:16.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.840 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.955008642, 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:16.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:16.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.95498728499999, 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:16.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.850 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.95231864100001, 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:16.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:16.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 183.94260106799999, 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:16.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:16.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 176.97649067899999, 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:16.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:16.868 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:16.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:16.870 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:16.869+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:16.970 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:16.970+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:17.008 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:17.008 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:17.008 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:17.008 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:17.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:17.071 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:17.071+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:17.114 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:17.114 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:17.115 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:17.115 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:17.115 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:17.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:17.172 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:17.172+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.221 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.551911137999994, 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:17.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.548898265000005, 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:17.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.546631340000005, 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.544461721999994, 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:17.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:17.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.541172355, 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.535666226000004, 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:17.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.532827940999994, 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:17.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.528684102, 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:17.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:17.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.514449245999998, 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:17.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:17.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.503994789999993, 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:17.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.476285419999996, 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:17.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.461744998, 2024-09-17T07:51:17.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.458011862999996, 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:17.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.446970179999994, 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:17.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.277 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.432595394000003, 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:17.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.418639519999999, 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:17.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:17.284 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.415005437000001, 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:17.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.409999577000001, 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.342719415000001, 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.292 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:17.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.333795346999999, 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:17.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.299 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:17.300 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:17.300 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:17.301 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:17.301+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:17.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:17.402 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:17.401+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:17.407 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:17.407 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:17.407 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:17.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:17.503 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:17.503+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:17.557 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:17.557 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:17.557 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:17.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:51:17.643 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:17.643+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.864988236999999, 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.863495214, 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.858509631, 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.854370108000001, 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:17.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.849585786999999, 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:17.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:17.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.841814540000001, 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:17.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.831776299000001, 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:17.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:17.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:17.703 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.812458739, 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.707 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.800060430999999, 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:17.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.796408875000001, 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:17.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:17.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.795825724, 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:17.719 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.791242382, 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.790913330999999, 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:17.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.782714134999999, 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 31.782192888000001, 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:17.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.151906081, 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:17.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:17.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.12755628, 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:17.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:17.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.744 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.126029470999999, 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:17.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.748 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.094784465, 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:17.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:17.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 28.042961406, 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:17.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:17.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:17.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:17.757 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:17.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:17.759 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:17.758+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:17.859 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:17.859 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:17.859 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:17.859 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:17.859 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:17.859+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:17.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:17.960 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:17.959+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:18.004 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:18.004 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:18.005 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:18.005 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:18.005 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:18.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:18.060 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:18.060+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:18.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.635041940000001, 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:18.153 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:18.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.619422274000001, 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:18.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.162 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.615341451999999, 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:18.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:18.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.608851309999999, 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:18.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.6031361, 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:18.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.595898465000001, 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.588367859999998, 2024-09-17T07:51:18.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:18.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 28.584592625999999, 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:18.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.185 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.63322582, 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:18.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:18.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:18.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.628516961999999, 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:18.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.62461991, 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:18.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.621030734000001, 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:18.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:18.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.204 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.616546416999999, 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:18.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:18.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:18.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.612614387000001, 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:18.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:18.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:18.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.608827548000001, 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.215 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:18.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.605175866, 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:18.219 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:18.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:18.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.601067425, 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:18.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:18.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.597467353999999, 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:18.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:18.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:18.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:18.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.594100761, 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:18.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.236 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.236 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:18.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:18.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:18.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:18.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:18.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:18.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:18.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:18.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:18.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:18.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.244 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.244 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 27.590709026999999, 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:18.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:18.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:18.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:18.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:18.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:18.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:18.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:18.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:18.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:18.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:18.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:18.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:18.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:18.253 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:18.253 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:18.254 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:18.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:18.255 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:18.254+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:18.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:18.260 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:18.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:18.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:18.276 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:18.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:18.292 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:18.308 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:18.308 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:18.323 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:18.324 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:18.360 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:18.360 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:18.360 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:18.359+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:18.368 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:18.456 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:18.456+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:18.510 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:18.510 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:18.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:18.557 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:18.557+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:18.661 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:18.661 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:18.661 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:18.660+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:18.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:18.762 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:18.762+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:18.812 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:18.812 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:18.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:18.863 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:18.863+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:18.962 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:18.962 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:18.963 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:18.963+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:18.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:19.064 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:19.064+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:19.113 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.113 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:19.165 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:19.165+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:19.229 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.229 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:19.282 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:19.282+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:19.336 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.336 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:19.387 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:19.386+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:19.468 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:19.468+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:19.483 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.483 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:19.569 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:19.569+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:19.589 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.589 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:19.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:51:19.682 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:19.682+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:19.737 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.737 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:19.786 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:19.786+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:19.872 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:19.872+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:19.884 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.884 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:19.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:19.973 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:19.972+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:19.995 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:19.995 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:20.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:20.087 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:20.087+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:20.144 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:20.145 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:20.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:20.175 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:20.175+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:20.276 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:20.276+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:20.295 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:20.296 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:20.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:20.338 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:20.382 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:20.382+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:20.403 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:20.404 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:20.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:20.454 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:20.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:20.469 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:20.470 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:20.478 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:20.478+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:20.484 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:20.484 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:20.499 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:20.500 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:20.512 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:20.512 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:20.515 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:20.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:20.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:20.531 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:20.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:20.582 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:20.582+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:20.592 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:51:20.661 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.143083349000001, 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.127463682999998, 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.123382861, 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.116892718999999, 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.111177509000001, 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.103939874000002, 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.096409268999999, 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.092634035, 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.141267229, 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.136558371, 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.132661319, 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.129072142999998, 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.124587825999999, 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.120655796000001, 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.116868957000001, 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.113217275, 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.109108834000001, 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.105508763, 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.10214217, 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 30.098750436, 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:20.745 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:20.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:20.747 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:20.746+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:20.847 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:20.847+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:20.849 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:20.849 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:20.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:20.948 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:20.948+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:20.999 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:21.000 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:21.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:21.049 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:21.049+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:21.150 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:21.150+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.984023248, 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:21.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.97636373200001, 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.967974164, 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:21.167 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.967602914, 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:21.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.959539949, 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.176 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.95916883699999, 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.94988843199999, 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.94822899900001, 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:21.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.94220417400001, 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.94195384700001, 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.934840009, 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.201 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.93432290999999, 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.915351743, 2024-09-17T07:51:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.90909488299999, 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.90255882100001, 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.52969241100001, 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.527732004, 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 192.64206336800001, 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 188.33121772999999, 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 188.32985628500001, 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.257 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:21.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:21.259 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:21.258+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:21.359 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:21.359+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:21.401 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:21.401 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:21.401 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:21.401 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:21.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:21.460 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:21.460+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:21.511 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:21.512 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:21.512 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:21.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:21.561 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:21.561+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:21.600 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:21.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 199.36592396099999, 2024-09-17T07:51:21.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:21.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.99761834500001, 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:21.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:21.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.30543609, 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:21.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.29957923399999, 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:21.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:21.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.28788627500001, 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.647 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:21.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:21.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:21.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.21903169199999, 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:21.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:21.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.207132403, 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:21.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:21.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:21.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.15628785600001, 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:21.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.150279649, 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.672 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:21.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.144579292, 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:21.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:21.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.13883966700001, 2024-09-17T07:51:21.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:21.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 188.80572861900001, 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:21.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:21.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:21.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.960627722, 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.696 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:21.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:21.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.95454099899999, 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.871084698, 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.708 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:21.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.86477026599999, 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:21.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.858398806, 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:21.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:21.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.85247633, 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.727 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:21.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:21.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:21.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 185.794493434, 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:21.733 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:21.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:21.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:21.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 181.836788762, 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:21.739 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:21.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:21.743 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:21.745 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:21.744+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:21.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:21.845 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:21.845+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:21.852 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:21.853 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:21.861 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:21.946 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:21.946+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:22.000 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:22.000 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:22.000 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:22.001 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:22.001 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:22.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:22.050 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:22.050+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.98179839599999, 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:22.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.153 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.97445906199999, 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.154 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:22.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.973741988, 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:22.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.163 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.96451442700001, 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:22.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.168 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.955289859, 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:22.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.939815219, 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.173 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:22.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:22.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.93052570899999, 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:22.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:22.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.92393546400001, 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:22.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.90775298299999, 2024-09-17T07:51:22.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:22.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:22.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.89867806300001, 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.192 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:22.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:22.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:22.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.196 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.558239187, 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:22.197 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:22.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:22.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.55618691000001, 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:22.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:22.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.495955259, 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:22.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:22.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:22.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.33477611000001, 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:22.219 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:22.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:22.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:22.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.33475180900001, 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.33471743300001, 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.334696076, 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:22.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.33202743199999, 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.240 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:22.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:22.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:22.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.322309859, 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:22.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:22.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 182.35619947000001, 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:22.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:22.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:22.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.257 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.257 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:22.257 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:22.148+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:22.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:22.259 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:22.260+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:22.360 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:22.360+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:22.397 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:22.397 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:22.397 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:22.397 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:22.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:22.461 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:22.461+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:22.503 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:22.503 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:22.503 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:22.504 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:22.504 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:22.512 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:22.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:22.586 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:22.586+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:22.600 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:51:22.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:51:22.611 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.942525270999994, 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.612 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:22.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:22.616 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.939512398000005, 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:22.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:22.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.937245473000004, 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:22.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.935075854999994, 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:22.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.931786488, 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:22.636 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.926280359000003, 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:22.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.923442073999993, 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:22.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:22.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.919298234999999, 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.645 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:22.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.905063378999998, 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:22.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.894608923000007, 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:22.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:22.656 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.866899552999996, 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.852359131, 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.663 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.848625995999996, 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.667 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.837584312999994, 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:22.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:22.671 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.823209527000003, 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:22.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.809253652999999, 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.675 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:22.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.805619569999998, 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:22.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.80061371, 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:22.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:22.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.733333547999997, 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:22.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.690 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.724409479999998, 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:22.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:22.694 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:22.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-17T07:51:22.697 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:22.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:22.698 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:22.696+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:22.698 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:22.699 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:22.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:22.714 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:22.714 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:22.732 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:22.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:22.748 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:22.749 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:22.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:22.764 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:22.816 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:22.816 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:22.816 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:22.816+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:22.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:22.898 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:22.898+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:22.968 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:22.968 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:22.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:22.999 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:22.999+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:23.078 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.078 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:23.131 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:23.131+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:23.187 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.187 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:23.236 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:23.236+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:23.302 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:23.302+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:23.333 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.333 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:23.403 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:23.403+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:23.479 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.479 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:23.504 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:23.503+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:23.594 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.594 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:23.604 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:23.604+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:23.607 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:51:23.607 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:51:23.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-17T07:51:23.705 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:23.705+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:23.706 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.707 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:23.805 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:23.805+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:23.858 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.859 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.867 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:23.910 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:23.909+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:23.968 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:23.969 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:23.977 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:24.018 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:24.017+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:24.108 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:24.107+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:24.119 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.119 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:24.212 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:24.211+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:24.269 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.269 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:24.318 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:24.317+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:24.388 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.388 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:24.410 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:24.409+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:24.496 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.497 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:24.511 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:24.510+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:24.603 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.603 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.608 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T07:51:24.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:24.613 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:24.612+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:24.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-17T07:51:24.713 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:24.711+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:24.750 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.750 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:24.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:24.821 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:24.820+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:24.856 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.856 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:24.894 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:24.910 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:24.915 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:24.915+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:24.925 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:24.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:24.940 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:24.941 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:24.956 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:24.956 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:24.963 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:24.963 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:24.971 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:24.971 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:24.972 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:25.024 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:25.023+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:25.075 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:25.075 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:25.075 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:25.075 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:25.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:25.124 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:25.123+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:25.218 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:25.218+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:25.225 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:25.225 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:25.225 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:25.226 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:25.226 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:25.234 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:25.319 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:25.319+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:25.337 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.531105984, 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.529612960999998, 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.524627377999998, 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.520487854999999, 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.515703533999996, 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.507932287000003, 2024-09-17T07:51:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.497894045999999, 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:25.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:25.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.478576486000001, 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.466178178, 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:25.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:25.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.462526621999999, 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:25.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.461943470999998, 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:25.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.457360129000001, 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:25.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.390 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.457031078, 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.391 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.392 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:25.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.448831882, 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:25.395 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.398 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.399 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 39.448310634999999, 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.400 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:25.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.402 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.818023828000001, 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:25.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.793674027000002, 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.792147217999997, 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.412 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:25.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.760902211999998, 2024-09-17T07:51:25.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:25.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:25.420 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 35.709079152999998, 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:25.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:25.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:25.425 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:25.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:25.428 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:25.427+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:25.527 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:25.527 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:25.527 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:25.527 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:25.527 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:25.526+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:25.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:25.612 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:51:25.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:51:25.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-17T07:51:25.633 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:25.633 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:25.634 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:25.634 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:25.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:25.667 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:25.666+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:25.729 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:25.728+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.224173325999999, 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.20855366, 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:25.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:25.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:25.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.204472838000001, 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:25.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.197982695999997, 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:25.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:25.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.192267485999999, 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:25.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.185029851000003, 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:25.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:25.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.177499245999996, 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:25.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 36.173724012000001, 2024-09-17T07:51:25.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:25.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.222357205999998, 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:25.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:25.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.217648347999997, 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.777 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:25.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:25.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:25.781 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.213751295999998, 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.21016212, 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:25.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:25.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.205677803, 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:25.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:25.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.201745772999999, 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.795 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:25.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.799 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.197958933999999, 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:25.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:25.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.803 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.194307252000002, 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:25.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.190198811000002, 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.186598740000001, 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.183232146999998, 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 35.179840413000001, 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:25.825 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:25.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:25.870 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:25.869+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:25.928 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:25.929 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:25.929 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:25.929 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:25.930 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:25.930+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:25.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:26.031 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:26.031+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:26.078 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:26.078 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:26.078 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:26.078 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:26.078 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:26.087 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:26.131 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:26.130+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.023601541, 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.01594202499999, 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:26.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.007552457, 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:26.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:26.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:26.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:26.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.007181207, 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.99911824200001, 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.99874713, 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:26.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:26.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.989466725, 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:26.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.98780729200001, 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:26.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:26.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:26.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:26.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.98178246699999, 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.226 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:26.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.230 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.98153214000001, 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:26.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:26.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.974418302, 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.973901203, 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:26.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:26.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.95493003600001, 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.247 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.948673176, 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:26.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.94213711399999, 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.56927070399999, 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.56731029700001, 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 197.68164166099999, 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.370796023, 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.36943457800001, 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.291 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:26.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:26.293 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:26.291+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:26.393 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:26.393+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:26.434 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:26.435 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:26.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:26.493 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:26.493+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:26.544 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:26.545 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:26.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:26.594 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:26.594+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:51:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:51:26.634 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.40126379099999, 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:26.664 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 199.032958175, 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.34077592, 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.33491906399999, 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.323226105, 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.25437152200001, 2024-09-17T07:51:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.242472233, 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.191627686, 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.185619479, 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.703 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:26.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:26.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.179919122, 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:26.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:26.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:26.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.17417949700001, 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:26.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 193.84106844900001, 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.995967552, 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:26.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:26.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.98988082899999, 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.906424528, 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.90011009599999, 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.89373863599999, 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:26.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.88781616, 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 190.829833264, 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:26.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:26.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 186.872128592, 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:26.772 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:26.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:26.774 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:26.773+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:26.874 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:26.874+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:26.884 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:26.885 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:26.885 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:26.885 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:26.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:26.975 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:26.974+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:26.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:27.033 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:27.033 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:27.033 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:27.034 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:27.034 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:27.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:27.086 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:27.086+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:27.099 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:27.115 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:27.130 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.142 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.97565229, 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:27.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.96831295600001, 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:27.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:27.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.96759588200001, 2024-09-17T07:51:27.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:27.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:27.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.95836832099999, 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:27.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.94914375299999, 2024-09-17T07:51:27.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:27.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:27.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.93366911300001, 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.92437960300001, 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.91778935799999, 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.90160687700001, 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.89253195699999, 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.55209308100001, 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.55004080399999, 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.48980915300001, 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.32863000399999, 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.32860570299999, 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.32857132699999, 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.32854997000001, 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.325881326, 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.31616375300001, 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 187.35005336399999, 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:27.241 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:27.242 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:27.242 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:27.243 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:27.243 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:27.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:27.244 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:27.242+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:27.244 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:27.244 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:27.343 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:27.342+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:27.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:27.444 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:27.443+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:27.487 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:27.487 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:27.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:27.545 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:27.545+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:27.594 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:27.594 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:27.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:27.623 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:51:27.638 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:51:27.638 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:27.646 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:27.647+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:27.713 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:27.713 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:27.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:27.763 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:27.762+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:27.847 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:27.847+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:27.865 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:27.865 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:27.874 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:27.948 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:27.947+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:27.955 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:27.955 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:27.968 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-17T07:51:27.621680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229742279,"num_objects":124,"num_object_clones":19,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":12,"num_read":5917,"num_read_kb":188110,"num_write":928,"num_write_kb":274751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":301990820,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":950,"ondisk_log_size":950,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2261060,"kb_used_data":449836,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":563969980,"statfs":{"total":579820584960,"available":577505259520,"internally_reserved":0,"allocated":460632064,"data_stored":465387464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016638"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":55,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588826+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:51:17.588826+0000","last_peered":"2024-09-17T07:51:17.588826+0000","last_clean":"2024-09-17T07:51:17.588826+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:51:17.588826+0000","last_undegraded":"2024-09-17T07:51:17.588826+0000","last_fullsized":"2024-09-17T07:51:17.588826+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"134'9","reported_seq":61,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221424+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:51:22.221424+0000","last_peered":"2024-09-17T07:51:22.221424+0000","last_clean":"2024-09-17T07:51:22.221424+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:22.221424+0000","last_undegraded":"2024-09-17T07:51:22.221424+0000","last_fullsized":"2024-09-17T07:51:22.221424+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777292,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":8,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"138'52","reported_seq":431,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612069+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:51:25.612069+0000","last_peered":"2024-09-17T07:51:25.612069+0000","last_clean":"2024-09-17T07:51:25.612069+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:51:25.612069+0000","last_undegraded":"2024-09-17T07:51:25.612069+0000","last_fullsized":"2024-09-17T07:51:25.612069+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:24.024613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16778394,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":1082,"num_read_kb":32099,"num_write":40,"num_write_kb":16451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":114,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612070+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:51:25.612070+0000","last_peered":"2024-09-17T07:51:25.612070+0000","last_clean":"2024-09-17T07:51:25.612070+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:51:25.612070+0000","last_undegraded":"2024-09-17T07:51:25.612070+0000","last_fullsized":"2024-09-17T07:51:25.612070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":125,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588824+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:51:17.588824+0000","last_peered":"2024-09-17T07:51:17.588824+0000","last_clean":"2024-09-17T07:51:17.588824+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:51:17.588824+0000","last_undegraded":"2024-09-17T07:51:17.588824+0000","last_fullsized":"2024-09-17T07:51:17.588824+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"139'405","reported_seq":1807,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989858+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:51:24.989858+0000","last_peered":"2024-09-17T07:51:24.989858+0000","last_clean":"2024-09-17T07:51:24.989858+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:24.989858+0000","last_undegraded":"2024-09-17T07:51:24.989858+0000","last_fullsized":"2024-09-17T07:51:24.989858+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686429,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":2009,"num_read_kb":84177,"num_write":490,"num_write_kb":66616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":141,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612210+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:51:25.612210+0000","last_peered":"2024-09-17T07:51:25.612210+0000","last_clean":"2024-09-17T07:51:25.612210+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:51:25.612210+0000","last_undegraded":"2024-09-17T07:51:25.612210+0000","last_fullsized":"2024-09-17T07:51:25.612210+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"138'6","reported_seq":37,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989850+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:51:24.989850+0000","last_peered":"2024-09-17T07:51:24.989850+0000","last_clean":"2024-09-17T07:51:24.989850+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:24.989850+0000","last_undegraded":"2024-09-17T07:51:24.989850+0000","last_fullsized":"2024-09-17T07:51:24.989850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582969,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":6,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"139'271","reported_seq":1414,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.811069+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:51:25.811069+0000","last_peered":"2024-09-17T07:51:25.811069+0000","last_clean":"2024-09-17T07:51:25.811069+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:25.811069+0000","last_undegraded":"2024-09-17T07:51:25.811069+0000","last_fullsized":"2024-09-17T07:51:25.811069+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:13.532835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18095381,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":2428,"num_read_kb":47021,"num_write":139,"num_write_kb":33826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221496+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:51:22.221496+0000","last_peered":"2024-09-17T07:51:22.221496+0000","last_clean":"2024-09-17T07:51:22.221496+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:51:22.221496+0000","last_undegraded":"2024-09-17T07:51:22.221496+0000","last_fullsized":"2024-09-17T07:51:22.221496+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612063+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:51:25.612063+0000","last_peered":"2024-09-17T07:51:25.612063+0000","last_clean":"2024-09-17T07:51:25.612063+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:51:25.612063+0000","last_undegraded":"2024-09-17T07:51:25.612063+0000","last_fullsized":"2024-09-17T07:51:25.612063+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":208,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612085+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:25.612085+0000","last_peered":"2024-09-17T07:51:25.612085+0000","last_clean":"2024-09-17T07:51:25.612085+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:25.612085+0000","last_undegraded":"2024-09-17T07:51:25.612085+0000","last_fullsized":"2024-09-17T07:51:25.612085+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"137'91","reported_seq":329,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.611806+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:51:25.611806+0000","last_peered":"2024-09-17T07:51:25.611806+0000","last_clean":"2024-09-17T07:51:25.611806+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:25.611806+0000","last_undegraded":"2024-09-17T07:51:25.611806+0000","last_fullsized":"2024-09-17T07:51:25.611806+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373608,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":222,"num_read_kb":24678,"num_write":98,"num_write_kb":25130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":28,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989841+0000","last_change":"2024-09-17T07:50:53.578257+0000","last_active":"2024-09-17T07:51:24.989841+0000","last_peered":"2024-09-17T07:51:24.989841+0000","last_clean":"2024-09-17T07:51:24.989841+0000","last_became_active":"2024-09-17T07:50:53.530399+0000","last_became_peered":"2024-09-17T07:50:53.530399+0000","last_unstale":"2024-09-17T07:51:24.989841+0000","last_undegraded":"2024-09-17T07:51:24.989841+0000","last_fullsized":"2024-09-17T07:51:24.989841+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:09.511077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"139'7","reported_seq":247,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612168+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:51:25.612168+0000","last_peered":"2024-09-17T07:51:25.612168+0000","last_clean":"2024-09-17T07:51:25.612168+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:25.612168+0000","last_undegraded":"2024-09-17T07:51:25.612168+0000","last_fullsized":"2024-09-17T07:51:25.612168+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":13,"num_write_kb":16393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":43,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989841+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:51:24.989841+0000","last_peered":"2024-09-17T07:51:24.989841+0000","last_clean":"2024-09-17T07:51:24.989841+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:51:24.989841+0000","last_undegraded":"2024-09-17T07:51:24.989841+0000","last_fullsized":"2024-09-17T07:51:24.989841+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":159,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612114+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:51:25.612114+0000","last_peered":"2024-09-17T07:51:25.612114+0000","last_clean":"2024-09-17T07:51:25.612114+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:25.612114+0000","last_undegraded":"2024-09-17T07:51:25.612114+0000","last_fullsized":"2024-09-17T07:51:25.612114+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":36,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221565+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:51:22.221565+0000","last_peered":"2024-09-17T07:51:22.221565+0000","last_clean":"2024-09-17T07:51:22.221565+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:51:22.221565+0000","last_undegraded":"2024-09-17T07:51:22.221565+0000","last_fullsized":"2024-09-17T07:51:22.221565+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":55,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221503+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:51:22.221503+0000","last_peered":"2024-09-17T07:51:22.221503+0000","last_clean":"2024-09-17T07:51:22.221503+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:22.221503+0000","last_undegraded":"2024-09-17T07:51:22.221503+0000","last_fullsized":"2024-09-17T07:51:22.221503+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":160,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612166+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:51:25.612166+0000","last_peered":"2024-09-17T07:51:25.612166+0000","last_clean":"2024-09-17T07:51:25.612166+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:51:25.612166+0000","last_undegraded":"2024-09-17T07:51:25.612166+0000","last_fullsized":"2024-09-17T07:51:25.612166+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":48,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588888+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:51:17.588888+0000","last_peered":"2024-09-17T07:51:17.588888+0000","last_clean":"2024-09-17T07:51:17.588888+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:51:17.588888+0000","last_undegraded":"2024-09-17T07:51:17.588888+0000","last_fullsized":"2024-09-17T07:51:17.588888+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":94933812,"num_objects":67,"num_object_clones":19,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":5741,"num_read_kb":187975,"num_write":767,"num_write_kb":142023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":6,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180305920,"data_stored":191155816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218029,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":75,"num_read_kb":51,"num_write":54,"num_write_kb":131094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":209715406,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268509184,"data_stored":268436058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860435,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369560,"kb_used_data":71436,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002280,"statfs":{"total":96636764160,"available":96258334720,"internally_reserved":0,"allocated":73150464,"data_stored":72123820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":4,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.26600000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363604,"kb_used_data":57920,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94008236,"statfs":{"total":96636764160,"available":96264433664,"internally_reserved":0,"allocated":59310080,"data_stored":65333861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38700000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34399999999999997}]}]},{"osd":3,"up_from":126,"seq":541165879305,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354052,"kb_used_data":56120,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94017788,"statfs":{"total":96636764160,"available":96274214912,"internally_reserved":0,"allocated":57466880,"data_stored":56429954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349976,"kb_used_data":44292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94021864,"statfs":{"total":96636764160,"available":96278388736,"internally_reserved":0,"allocated":45355008,"data_stored":44311327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.308}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38200000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.36199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640315,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411536,"kb_used_data":105860,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93960304,"statfs":{"total":96636764160,"available":96215351296,"internally_reserved":0,"allocated":108400640,"data_stored":107350324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.46999999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48699999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.50700000000000001}]}]},{"osd":0,"up_from":121,"seq":519691042825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412332,"kb_used_data":114208,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93959508,"statfs":{"total":96636764160,"available":96214536192,"internally_reserved":0,"allocated":116948992,"data_stored":119838178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48078848,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30519296,"data_stored":30486959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":30952893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13127680,"data_stored":13115560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44933120,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:51:27.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:27.978 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:27.978 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:27.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:28.050 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:28.048+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:28.086 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.086 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:28.150 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:28.150+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:28.192 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.193 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:28.250 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:28.252 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:28.250+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:28.262 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-17T07:51:27.621680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229742279,"num_objects":124,"num_object_clones":19,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":12,"num_read":5917,"num_read_kb":188110,"num_write":928,"num_write_kb":274751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":301990820,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":950,"ondisk_log_size":950,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2261060,"kb_used_data":449836,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":563969980,"statfs":{"total":579820584960,"available":577505259520,"internally_reserved":0,"allocated":460632064,"data_stored":465387464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016638"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":55,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588826+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:51:17.588826+0000","last_peered":"2024-09-17T07:51:17.588826+0000","last_clean":"2024-09-17T07:51:17.588826+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:51:17.588826+0000","last_undegraded":"2024-09-17T07:51:17.588826+0000","last_fullsized":"2024-09-17T07:51:17.588826+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"134'9","reported_seq":61,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221424+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:51:22.221424+0000","last_peered":"2024-09-17T07:51:22.221424+0000","last_clean":"2024-09-17T07:51:22.221424+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:22.221424+0000","last_undegraded":"2024-09-17T07:51:22.221424+0000","last_fullsized":"2024-09-17T07:51:22.221424+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777292,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":8,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"138'52","reported_seq":431,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612069+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:51:25.612069+0000","last_peered":"2024-09-17T07:51:25.612069+0000","last_clean":"2024-09-17T07:51:25.612069+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:51:25.612069+0000","last_undegraded":"2024-09-17T07:51:25.612069+0000","last_fullsized":"2024-09-17T07:51:25.612069+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:24.024613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16778394,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":1082,"num_read_kb":32099,"num_write":40,"num_write_kb":16451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":114,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612070+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:51:25.612070+0000","last_peered":"2024-09-17T07:51:25.612070+0000","last_clean":"2024-09-17T07:51:25.612070+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:51:25.612070+0000","last_undegraded":"2024-09-17T07:51:25.612070+0000","last_fullsized":"2024-09-17T07:51:25.612070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":125,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588824+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:51:17.588824+0000","last_peered":"2024-09-17T07:51:17.588824+0000","last_clean":"2024-09-17T07:51:17.588824+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:51:17.588824+0000","last_undegraded":"2024-09-17T07:51:17.588824+0000","last_fullsized":"2024-09-17T07:51:17.588824+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"139'405","reported_seq":1807,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989858+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:51:24.989858+0000","last_peered":"2024-09-17T07:51:24.989858+0000","last_clean":"2024-09-17T07:51:24.989858+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:24.989858+0000","last_undegraded":"2024-09-17T07:51:24.989858+0000","last_fullsized":"2024-09-17T07:51:24.989858+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686429,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":2009,"num_read_kb":84177,"num_write":490,"num_write_kb":66616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":141,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612210+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:51:25.612210+0000","last_peered":"2024-09-17T07:51:25.612210+0000","last_clean":"2024-09-17T07:51:25.612210+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:51:25.612210+0000","last_undegraded":"2024-09-17T07:51:25.612210+0000","last_fullsized":"2024-09-17T07:51:25.612210+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"138'6","reported_seq":37,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989850+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:51:24.989850+0000","last_peered":"2024-09-17T07:51:24.989850+0000","last_clean":"2024-09-17T07:51:24.989850+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:24.989850+0000","last_undegraded":"2024-09-17T07:51:24.989850+0000","last_fullsized":"2024-09-17T07:51:24.989850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582969,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":6,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"139'271","reported_seq":1414,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.811069+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:51:25.811069+0000","last_peered":"2024-09-17T07:51:25.811069+0000","last_clean":"2024-09-17T07:51:25.811069+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:25.811069+0000","last_undegraded":"2024-09-17T07:51:25.811069+0000","last_fullsized":"2024-09-17T07:51:25.811069+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:13.532835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18095381,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":2428,"num_read_kb":47021,"num_write":139,"num_write_kb":33826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221496+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:51:22.221496+0000","last_peered":"2024-09-17T07:51:22.221496+0000","last_clean":"2024-09-17T07:51:22.221496+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:51:22.221496+0000","last_undegraded":"2024-09-17T07:51:22.221496+0000","last_fullsized":"2024-09-17T07:51:22.221496+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612063+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:51:25.612063+0000","last_peered":"2024-09-17T07:51:25.612063+0000","last_clean":"2024-09-17T07:51:25.612063+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:51:25.612063+0000","last_undegraded":"2024-09-17T07:51:25.612063+0000","last_fullsized":"2024-09-17T07:51:25.612063+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":208,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612085+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:25.612085+0000","last_peered":"2024-09-17T07:51:25.612085+0000","last_clean":"2024-09-17T07:51:25.612085+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:25.612085+0000","last_undegraded":"2024-09-17T07:51:25.612085+0000","last_fullsized":"2024-09-17T07:51:25.612085+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"137'91","reported_seq":329,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.611806+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:51:25.611806+0000","last_peered":"2024-09-17T07:51:25.611806+0000","last_clean":"2024-09-17T07:51:25.611806+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:25.611806+0000","last_undegraded":"2024-09-17T07:51:25.611806+0000","last_fullsized":"2024-09-17T07:51:25.611806+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373608,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":222,"num_read_kb":24678,"num_write":98,"num_write_kb":25130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":28,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989841+0000","last_change":"2024-09-17T07:50:53.578257+0000","last_active":"2024-09-17T07:51:24.989841+0000","last_peered":"2024-09-17T07:51:24.989841+0000","last_clean":"2024-09-17T07:51:24.989841+0000","last_became_active":"2024-09-17T07:50:53.530399+0000","last_became_peered":"2024-09-17T07:50:53.530399+0000","last_unstale":"2024-09-17T07:51:24.989841+0000","last_undegraded":"2024-09-17T07:51:24.989841+0000","last_fullsized":"2024-09-17T07:51:24.989841+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:09.511077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"139'7","reported_seq":247,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612168+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:51:25.612168+0000","last_peered":"2024-09-17T07:51:25.612168+0000","last_clean":"2024-09-17T07:51:25.612168+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:25.612168+0000","last_undegraded":"2024-09-17T07:51:25.612168+0000","last_fullsized":"2024-09-17T07:51:25.612168+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":13,"num_write_kb":16393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":43,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989841+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:51:24.989841+0000","last_peered":"2024-09-17T07:51:24.989841+0000","last_clean":"2024-09-17T07:51:24.989841+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:51:24.989841+0000","last_undegraded":"2024-09-17T07:51:24.989841+0000","last_fullsized":"2024-09-17T07:51:24.989841+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":159,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612114+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:51:25.612114+0000","last_peered":"2024-09-17T07:51:25.612114+0000","last_clean":"2024-09-17T07:51:25.612114+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:25.612114+0000","last_undegraded":"2024-09-17T07:51:25.612114+0000","last_fullsized":"2024-09-17T07:51:25.612114+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":36,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221565+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:51:22.221565+0000","last_peered":"2024-09-17T07:51:22.221565+0000","last_clean":"2024-09-17T07:51:22.221565+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:51:22.221565+0000","last_undegraded":"2024-09-17T07:51:22.221565+0000","last_fullsized":"2024-09-17T07:51:22.221565+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":55,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221503+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:51:22.221503+0000","last_peered":"2024-09-17T07:51:22.221503+0000","last_clean":"2024-09-17T07:51:22.221503+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:22.221503+0000","last_undegraded":"2024-09-17T07:51:22.221503+0000","last_fullsized":"2024-09-17T07:51:22.221503+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":160,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612166+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:51:25.612166+0000","last_peered":"2024-09-17T07:51:25.612166+0000","last_clean":"2024-09-17T07:51:25.612166+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:51:25.612166+0000","last_undegraded":"2024-09-17T07:51:25.612166+0000","last_fullsized":"2024-09-17T07:51:25.612166+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":48,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588888+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:51:17.588888+0000","last_peered":"2024-09-17T07:51:17.588888+0000","last_clean":"2024-09-17T07:51:17.588888+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:51:17.588888+0000","last_undegraded":"2024-09-17T07:51:17.588888+0000","last_fullsized":"2024-09-17T07:51:17.588888+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":94933812,"num_objects":67,"num_object_clones":19,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":5741,"num_read_kb":187975,"num_write":767,"num_write_kb":142023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":6,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180305920,"data_stored":191155816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218029,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":75,"num_read_kb":51,"num_write":54,"num_write_kb":131094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":209715406,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268509184,"data_stored":268436058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860435,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369560,"kb_used_data":71436,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002280,"statfs":{"total":96636764160,"available":96258334720,"internally_reserved":0,"allocated":73150464,"data_stored":72123820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":4,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.26600000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363604,"kb_used_data":57920,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94008236,"statfs":{"total":96636764160,"available":96264433664,"internally_reserved":0,"allocated":59310080,"data_stored":65333861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38700000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34399999999999997}]}]},{"osd":3,"up_from":126,"seq":541165879305,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354052,"kb_used_data":56120,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94017788,"statfs":{"total":96636764160,"available":96274214912,"internally_reserved":0,"allocated":57466880,"data_stored":56429954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349976,"kb_used_data":44292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94021864,"statfs":{"total":96636764160,"available":96278388736,"internally_reserved":0,"allocated":45355008,"data_stored":44311327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.308}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38200000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.36199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640315,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411536,"kb_used_data":105860,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93960304,"statfs":{"total":96636764160,"available":96215351296,"internally_reserved":0,"allocated":108400640,"data_stored":107350324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.46999999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48699999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.50700000000000001}]}]},{"osd":0,"up_from":121,"seq":519691042825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412332,"kb_used_data":114208,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93959508,"statfs":{"total":96636764160,"available":96214536192,"internally_reserved":0,"allocated":116948992,"data_stored":119838178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48078848,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30519296,"data_stored":30486959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":30952893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13127680,"data_stored":13115560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44933120,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:51:28.263 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-17T07:51:28.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:28.264 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T07:51:28.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:51:28.339 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.339 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.348 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:28.353 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:28.353+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:28.453 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:28.452+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:28.466 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.466 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:28.555 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:28.555+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:28.599 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 150, 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:51:27.617818+0000", 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:50:48.503993+0000", 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:50:32.754467+0000", 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:51:28.600 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 7, 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:51:28.601 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:28.602 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:28.603 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "138", 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 8, 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 138, 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:28.604 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:28.605 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:28.606 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:51:28.607 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:28.608 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:28.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "138", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 8, 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 138, 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "new_default_pool27356", 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:48:02.539579+0000", 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:51:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "49", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 3, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 49, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:28.622 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 38, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 121, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/69763", 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/69763", 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/69763", 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/69763", 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:51:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 127, 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 49, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 126, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:51:28.641 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:28.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/61678", 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/61678", 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/61678", 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/61678", 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:51:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.649 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.650 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:51:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 68, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/57427", 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/57427", 2024-09-17T07:51:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/57427", 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/57427", 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:51:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "4.0", 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [ 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.7", 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: "mappings": [ 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:28.664 INFO:teuthology.orchestra.run.smithi026.stdout: "from": 2, 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "to": 5 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:51:28.665 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:51:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:51:28.667 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:51:28.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:28.667 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:28.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:51:28.669 INFO:tasks.thrashosds.thrasher:j is {'epoch': 150, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:51:27.617818+0000', 'last_up_change': '2024-09-17T07:50:48.503993+0000', 'last_in_change': '2024-09-17T07:50:32.754467+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool27356', 'create_time': '2024-09-17T07:48:02.539579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 38, 'up_from': 121, 'up_thru': 129, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 69763}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 69763}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 69763}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 69763}]}, 'public_addr': '172.21.15.26:6801/69763', 'cluster_addr': '172.21.15.26:6803/69763', 'heartbeat_back_addr': '172.21.15.26:6807/69763', 'heartbeat_front_addr': '172.21.15.26:6805/69763', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 11, 'last_clean_end': 49, 'up_from': 126, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 61678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 61678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 61678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 61678}]}, 'public_addr': '172.21.15.94:6817/61678', 'cluster_addr': '172.21.15.94:6819/61678', 'heartbeat_back_addr': '172.21.15.94:6823/61678', 'heartbeat_front_addr': '172.21.15.94:6821/61678', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 68, 'up_from': 96, 'up_thru': 129, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 57427}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 57427}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 57427}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 57427}]}, 'public_addr': '172.21.15.94:6801/57427', 'cluster_addr': '172.21.15.94:6803/57427', 'heartbeat_back_addr': '172.21.15.94:6807/57427', 'heartbeat_front_addr': '172.21.15.94:6805/57427', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}, {'pgid': '3.1', 'osds': [3, 2]}, {'pgid': '4.0', 'osds': [4, 5]}], 'pg_upmap_items': [{'pgid': '1.7', 'mappings': [{'from': 2, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:51:28.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:28.670 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.670 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:28.671 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:28.667+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:28.769 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:28.768+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:28.817 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.817 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:28.871 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:28.870+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:28.970 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:28.970 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:28.971 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:28.970+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:28.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:28.988 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:28.988 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:29.000 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-17T07:51:27.621680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229742279,"num_objects":124,"num_object_clones":19,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":12,"num_read":5917,"num_read_kb":188110,"num_write":928,"num_write_kb":274751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":301990820,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":950,"ondisk_log_size":950,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2261060,"kb_used_data":449836,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":563969980,"statfs":{"total":579820584960,"available":577505259520,"internally_reserved":0,"allocated":460632064,"data_stored":465387464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016638"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":55,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588826+0000","last_change":"2024-09-17T07:50:52.528092+0000","last_active":"2024-09-17T07:51:17.588826+0000","last_peered":"2024-09-17T07:51:17.588826+0000","last_clean":"2024-09-17T07:51:17.588826+0000","last_became_active":"2024-09-17T07:50:52.527794+0000","last_became_peered":"2024-09-17T07:50:52.527794+0000","last_unstale":"2024-09-17T07:51:17.588826+0000","last_undegraded":"2024-09-17T07:51:17.588826+0000","last_fullsized":"2024-09-17T07:51:17.588826+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:12.970848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"134'9","reported_seq":61,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221424+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:51:22.221424+0000","last_peered":"2024-09-17T07:51:22.221424+0000","last_clean":"2024-09-17T07:51:22.221424+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:22.221424+0000","last_undegraded":"2024-09-17T07:51:22.221424+0000","last_fullsized":"2024-09-17T07:51:22.221424+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777292,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":8,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"138'52","reported_seq":431,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612069+0000","last_change":"2024-09-17T07:50:33.872495+0000","last_active":"2024-09-17T07:51:25.612069+0000","last_peered":"2024-09-17T07:51:25.612069+0000","last_clean":"2024-09-17T07:51:25.612069+0000","last_became_active":"2024-09-17T07:50:33.767158+0000","last_became_peered":"2024-09-17T07:50:33.767158+0000","last_unstale":"2024-09-17T07:51:25.612069+0000","last_undegraded":"2024-09-17T07:51:25.612069+0000","last_fullsized":"2024-09-17T07:51:25.612069+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:24.024613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16778394,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":1082,"num_read_kb":32099,"num_write":40,"num_write_kb":16451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":25166056,"num_keys_recovered":6,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":114,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612070+0000","last_change":"2024-09-17T07:50:17.505685+0000","last_active":"2024-09-17T07:51:25.612070+0000","last_peered":"2024-09-17T07:51:25.612070+0000","last_clean":"2024-09-17T07:51:25.612070+0000","last_became_active":"2024-09-17T07:50:17.505432+0000","last_became_peered":"2024-09-17T07:50:17.505432+0000","last_unstale":"2024-09-17T07:51:25.612070+0000","last_undegraded":"2024-09-17T07:51:25.612070+0000","last_fullsized":"2024-09-17T07:51:25.612070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:55:30.828381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":125,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588824+0000","last_change":"2024-09-17T07:50:45.907051+0000","last_active":"2024-09-17T07:51:17.588824+0000","last_peered":"2024-09-17T07:51:17.588824+0000","last_clean":"2024-09-17T07:51:17.588824+0000","last_became_active":"2024-09-17T07:50:45.802650+0000","last_became_peered":"2024-09-17T07:50:45.802650+0000","last_unstale":"2024-09-17T07:51:17.588824+0000","last_undegraded":"2024-09-17T07:51:17.588824+0000","last_fullsized":"2024-09-17T07:51:17.588824+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:56.606694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"139'405","reported_seq":1807,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989858+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:51:24.989858+0000","last_peered":"2024-09-17T07:51:24.989858+0000","last_clean":"2024-09-17T07:51:24.989858+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:24.989858+0000","last_undegraded":"2024-09-17T07:51:24.989858+0000","last_fullsized":"2024-09-17T07:51:24.989858+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:12:22.813944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686429,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":2009,"num_read_kb":84177,"num_write":490,"num_write_kb":66616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":141,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612210+0000","last_change":"2024-09-17T07:50:34.767175+0000","last_active":"2024-09-17T07:51:25.612210+0000","last_peered":"2024-09-17T07:51:25.612210+0000","last_clean":"2024-09-17T07:51:25.612210+0000","last_became_active":"2024-09-17T07:50:34.766835+0000","last_became_peered":"2024-09-17T07:50:34.766835+0000","last_unstale":"2024-09-17T07:51:25.612210+0000","last_undegraded":"2024-09-17T07:51:25.612210+0000","last_fullsized":"2024-09-17T07:51:25.612210+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:43.977903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"138'6","reported_seq":37,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989850+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:51:24.989850+0000","last_peered":"2024-09-17T07:51:24.989850+0000","last_clean":"2024-09-17T07:51:24.989850+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:24.989850+0000","last_undegraded":"2024-09-17T07:51:24.989850+0000","last_fullsized":"2024-09-17T07:51:24.989850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582969,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":6,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"139'271","reported_seq":1414,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.811069+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:51:25.811069+0000","last_peered":"2024-09-17T07:51:25.811069+0000","last_clean":"2024-09-17T07:51:25.811069+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:25.811069+0000","last_undegraded":"2024-09-17T07:51:25.811069+0000","last_fullsized":"2024-09-17T07:51:25.811069+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:13.532835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18095381,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":2428,"num_read_kb":47021,"num_write":139,"num_write_kb":33826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221496+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:51:22.221496+0000","last_peered":"2024-09-17T07:51:22.221496+0000","last_clean":"2024-09-17T07:51:22.221496+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:51:22.221496+0000","last_undegraded":"2024-09-17T07:51:22.221496+0000","last_fullsized":"2024-09-17T07:51:22.221496+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"55'11","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612063+0000","last_change":"2024-09-17T07:50:34.894626+0000","last_active":"2024-09-17T07:51:25.612063+0000","last_peered":"2024-09-17T07:51:25.612063+0000","last_clean":"2024-09-17T07:51:25.612063+0000","last_became_active":"2024-09-17T07:50:34.766938+0000","last_became_peered":"2024-09-17T07:50:34.766938+0000","last_unstale":"2024-09-17T07:51:25.612063+0000","last_undegraded":"2024-09-17T07:51:25.612063+0000","last_fullsized":"2024-09-17T07:51:25.612063+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:40.881673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748774,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":10,"num_write_kb":36865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":208,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612085+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:25.612085+0000","last_peered":"2024-09-17T07:51:25.612085+0000","last_clean":"2024-09-17T07:51:25.612085+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:25.612085+0000","last_undegraded":"2024-09-17T07:51:25.612085+0000","last_fullsized":"2024-09-17T07:51:25.612085+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"137'91","reported_seq":329,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.611806+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:51:25.611806+0000","last_peered":"2024-09-17T07:51:25.611806+0000","last_clean":"2024-09-17T07:51:25.611806+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:25.611806+0000","last_undegraded":"2024-09-17T07:51:25.611806+0000","last_fullsized":"2024-09-17T07:51:25.611806+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373608,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":222,"num_read_kb":24678,"num_write":98,"num_write_kb":25130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":28,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989841+0000","last_change":"2024-09-17T07:50:53.578257+0000","last_active":"2024-09-17T07:51:24.989841+0000","last_peered":"2024-09-17T07:51:24.989841+0000","last_clean":"2024-09-17T07:51:24.989841+0000","last_became_active":"2024-09-17T07:50:53.530399+0000","last_became_peered":"2024-09-17T07:50:53.530399+0000","last_unstale":"2024-09-17T07:51:24.989841+0000","last_undegraded":"2024-09-17T07:51:24.989841+0000","last_fullsized":"2024-09-17T07:51:24.989841+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:09.511077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":35,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"139'7","reported_seq":247,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612168+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:51:25.612168+0000","last_peered":"2024-09-17T07:51:25.612168+0000","last_clean":"2024-09-17T07:51:25.612168+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:25.612168+0000","last_undegraded":"2024-09-17T07:51:25.612168+0000","last_fullsized":"2024-09-17T07:51:25.612168+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":13,"num_write_kb":16393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":43,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-17T07:51:24.989841+0000","last_change":"2024-09-17T07:50:34.769044+0000","last_active":"2024-09-17T07:51:24.989841+0000","last_peered":"2024-09-17T07:51:24.989841+0000","last_clean":"2024-09-17T07:51:24.989841+0000","last_became_active":"2024-09-17T07:50:34.765852+0000","last_became_peered":"2024-09-17T07:50:34.765852+0000","last_unstale":"2024-09-17T07:51:24.989841+0000","last_undegraded":"2024-09-17T07:51:24.989841+0000","last_fullsized":"2024-09-17T07:51:24.989841+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:47.610807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":159,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612114+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:51:25.612114+0000","last_peered":"2024-09-17T07:51:25.612114+0000","last_clean":"2024-09-17T07:51:25.612114+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:25.612114+0000","last_undegraded":"2024-09-17T07:51:25.612114+0000","last_fullsized":"2024-09-17T07:51:25.612114+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":36,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221565+0000","last_change":"2024-09-17T07:50:45.895486+0000","last_active":"2024-09-17T07:51:22.221565+0000","last_peered":"2024-09-17T07:51:22.221565+0000","last_clean":"2024-09-17T07:51:22.221565+0000","last_became_active":"2024-09-17T07:50:45.804779+0000","last_became_peered":"2024-09-17T07:50:45.804779+0000","last_unstale":"2024-09-17T07:51:22.221565+0000","last_undegraded":"2024-09-17T07:51:22.221565+0000","last_fullsized":"2024-09-17T07:51:22.221565+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:08.317956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":55,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-17T07:51:22.221503+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:51:22.221503+0000","last_peered":"2024-09-17T07:51:22.221503+0000","last_clean":"2024-09-17T07:51:22.221503+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:22.221503+0000","last_undegraded":"2024-09-17T07:51:22.221503+0000","last_fullsized":"2024-09-17T07:51:22.221503+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":160,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-17T07:51:25.612166+0000","last_change":"2024-09-17T07:50:33.766835+0000","last_active":"2024-09-17T07:51:25.612166+0000","last_peered":"2024-09-17T07:51:25.612166+0000","last_clean":"2024-09-17T07:51:25.612166+0000","last_became_active":"2024-09-17T07:50:33.766511+0000","last_became_peered":"2024-09-17T07:50:33.766511+0000","last_unstale":"2024-09-17T07:51:25.612166+0000","last_undegraded":"2024-09-17T07:51:25.612166+0000","last_fullsized":"2024-09-17T07:51:25.612166+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:21.366867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":48,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-17T07:51:17.588888+0000","last_change":"2024-09-17T07:50:33.815012+0000","last_active":"2024-09-17T07:51:17.588888+0000","last_peered":"2024-09-17T07:51:17.588888+0000","last_clean":"2024-09-17T07:51:17.588888+0000","last_became_active":"2024-09-17T07:50:33.766991+0000","last_became_peered":"2024-09-17T07:50:33.766991+0000","last_unstale":"2024-09-17T07:51:17.588888+0000","last_undegraded":"2024-09-17T07:51:17.588888+0000","last_fullsized":"2024-09-17T07:51:17.588888+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:06.012012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":67,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":94933812,"num_objects":67,"num_object_clones":19,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":5741,"num_read_kb":187975,"num_write":767,"num_write_kb":142023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":92275347,"num_keys_recovered":6,"num_objects_omap":6,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":44,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180305920,"data_stored":191155816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218029,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":75,"num_read_kb":51,"num_write":54,"num_write_kb":131094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":209715406,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268509184,"data_stored":268436058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860435,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369560,"kb_used_data":71436,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94002280,"statfs":{"total":96636764160,"available":96258334720,"internally_reserved":0,"allocated":73150464,"data_stored":72123820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":4,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.26600000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363604,"kb_used_data":57920,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94008236,"statfs":{"total":96636764160,"available":96264433664,"internally_reserved":0,"allocated":59310080,"data_stored":65333861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38700000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34399999999999997}]}]},{"osd":3,"up_from":126,"seq":541165879305,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354052,"kb_used_data":56120,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94017788,"statfs":{"total":96636764160,"available":96274214912,"internally_reserved":0,"allocated":57466880,"data_stored":56429954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349976,"kb_used_data":44292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94021864,"statfs":{"total":96636764160,"available":96278388736,"internally_reserved":0,"allocated":45355008,"data_stored":44311327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.308}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38200000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.36199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640315,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411536,"kb_used_data":105860,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93960304,"statfs":{"total":96636764160,"available":96215351296,"internally_reserved":0,"allocated":108400640,"data_stored":107350324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.46999999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48699999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.50700000000000001}]}]},{"osd":0,"up_from":121,"seq":519691042825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412332,"kb_used_data":114208,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93959508,"statfs":{"total":96636764160,"available":96214536192,"internally_reserved":0,"allocated":116948992,"data_stored":119838178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67133440,"data_stored":67109073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75501568,"data_stored":75497510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48078848,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30519296,"data_stored":30486959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":30952893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13127680,"data_stored":13115560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44933120,"data_stored":51975389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:51:29.003 INFO:tasks.thrashosds.thrasher:Setting 4.4 to [3, 4] 2024-09-17T07:51:29.004 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.4', '3', '4'] 2024-09-17T07:51:29.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.4 3 4 2024-09-17T07:51:29.072 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:29.072+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:29.090 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:29.090 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:29.099 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:29.173 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:29.171+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:29.203 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:29.203 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:29.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:29.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:29.296 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:29.295+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:29.320 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:29.320 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:29.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:29.368 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:29.369 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:29.375 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:29.375+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:29.427 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:29.427 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:29.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:29.477 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:29.477+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:29.573 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:29.573 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:29.576 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:29.575+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:29.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:29.585 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:29.600 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:29.600 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:29.616 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:29.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:51:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:set 4.4 pg_upmap mapping to [3,4] 2024-09-17T07:51:29.637 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:29.637 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:29.656 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:29.682 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:29.682 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:29.682 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:29.682 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:29.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:29.707 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:29.706+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:29.778 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:29.776+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:29.789 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:29.789 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:29.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:29.879 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:29.878+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.227605744000002, 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:29.897 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:29.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.224592870999999, 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.222325945999998, 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.220156328000002, 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.216866960999994, 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.917 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.211360831999997, 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:29.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:29.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.208522547000001, 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.204378707999993, 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.926 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:29.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.190143852000006, 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:29.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.179689396000001, 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:29.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.937 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.151980026000004, 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.941 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.137439603999994, 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:29.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.133706469000003, 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.948 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.122664786000001, 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:29.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:29.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:29.952 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.108289999999997, 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:29.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 84.094334126000007, 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.956 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:29.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.090700042999998, 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.085694183000001, 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.018414020999998, 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:29.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.970 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 44.009489952999999, 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:29.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:29.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:29.974 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:29.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:30.016 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:30.016+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:30.081 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:30.081+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:30.082 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:30.082 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:30.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:30.182 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:30.182+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:30.193 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:30.193 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:30.193 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:30.193 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:30.193 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:30.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:30.284 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:30.284+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:30.304 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:30.304 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:30.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.497824946000001, 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:30.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:30.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.496331923, 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:30.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:30.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.49134634, 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:30.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.317 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.487206817000001, 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:30.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.482422495999998, 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:30.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:30.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.474651248999997, 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.464613008000001, 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.333 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.445295448000003, 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:30.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:30.337 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.432897140000001, 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:30.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:30.341 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.429245584, 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:30.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:30.345 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.428662433, 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:30.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.424079091000003, 2024-09-17T07:51:30.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.423750040000002, 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.415550844000002, 2024-09-17T07:51:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 44.415029597, 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:30.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:30.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.784742790000003, 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:30.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.760392989000003, 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.373 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.758866179999998, 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:30.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:30.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.727621173999999, 2024-09-17T07:51:30.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:30.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:30.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 40.675798114999999, 2024-09-17T07:51:30.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:30.383 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:30.385 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:30.388 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:30.389 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:30.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:30.391 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:30.390+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:30.491 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:30.491 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:30.491 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:30.492 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:30.491+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:30.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:30.592 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:30.592+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:30.597 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:30.597 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:30.598 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:30.598 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:30.598 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:30.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:30.693 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:30.693+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.228312428999999, 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:30.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.212692763, 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:30.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:30.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.756 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.208611941000001, 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:30.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.760 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.202121798999997, 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.196406588999999, 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.764 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:30.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.189168954000003, 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:30.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:30.771 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.181638349000004, 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:30.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:30.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.177863115000001, 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:30.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.226496308999998, 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:30.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.221787450999997, 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:30.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.217890398999998, 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:30.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:30.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:30.791 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.214301223, 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.209816906, 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.205884875999999, 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.800 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:30.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:30.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.202098036999999, 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:30.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:30.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.198446355000002, 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:30.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:30.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.194337914000002, 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:30.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.190737843000001, 2024-09-17T07:51:30.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.821 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.187371249999998, 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.183979516000001, 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:30.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:30.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:30.829 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:30.830 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:30.831 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:30.831+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:30.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:30.931 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:30.930+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:30.935 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:30.936 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:30.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:31.032 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:31.032+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:31.086 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:31.086 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:31.086 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:31.086 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:31.086 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:31.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:31.138 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:31.137+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:31.197 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:31.197 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.03081137999999, 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.198 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:31.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:31.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.023151864, 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:31.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:31.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.211 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.01476229599999, 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:31.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:31.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.217 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.01439104599999, 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:31.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:31.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.00632808099999, 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.222 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.00595696900001, 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.99667656400001, 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.995017131, 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.988992306, 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.988741979, 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:31.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:31.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:31.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.98162814099999, 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:31.257 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.98111104200001, 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:31.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:31.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.96213987499999, 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:31.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:31.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:31.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.95588301500001, 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.268 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:31.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.949346953, 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:31.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.576480543, 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.278 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:31.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:31.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:31.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.57452013599999, 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:31.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:31.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:31.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 202.6888515, 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:31.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:31.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.37800586200001, 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:31.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:31.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:31.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.302 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.37664441699999, 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.308 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:31.310 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:31.310 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:31.309+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:31.410 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:31.409+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:31.453 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:31.454 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:31.454 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:31.454 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:31.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:31.511 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:31.511+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:31.605 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:31.605 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:31.605 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:31.605 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:31.605 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:31.612 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:31.612+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:31.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:31.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:51:31.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:31.712 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:31.712+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.464457084, 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.09615146799999, 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.40396921300001, 2024-09-17T07:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.398112357, 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.38641939799999, 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.317564815, 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:31.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:31.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.756 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.30566552600001, 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.757 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:31.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:31.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.25482097899999, 2024-09-17T07:51:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.24881277200001, 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.24311241500001, 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.23737278999999, 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.780 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:31.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:31.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:31.785 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 198.90426174199999, 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:31.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:31.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:31.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 196.05916084500001, 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:31.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:31.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.797 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 196.053074122, 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:31.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:31.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 195.96961782100001, 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:31.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.809 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 195.963303389, 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:31.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:31.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 195.95693192900001, 2024-09-17T07:51:31.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:31.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:31.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 195.95100945300001, 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:31.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 195.89302655700001, 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 191.93532188500001, 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:31.839 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:31.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:31.841 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:31.842 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:31.839+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:31.842 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:31.855 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:31.871 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:31.887 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:31.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:31.903 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:31.951 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:31.952 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:31.954 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:31.953+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:31.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:32.042 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:32.041+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:32.060 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.060 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:32.143 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:32.143+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:32.166 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.166 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:32.244 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:32.244+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:32.272 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.273 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:32.346 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:32.345+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:32.418 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.419 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:32.470 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:32.470+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:32.524 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.524 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:32.576 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:32.575+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:51:32.642 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.642 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.649 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:32.647+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:32.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:32.749 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:32.750+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:32.794 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.795 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:32.850 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:32.849+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:32.905 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:32.905 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:32.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:32.954 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:32.954+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:33.014 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.014 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:33.052 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:33.051+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:33.120 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.120 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.128 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:33.153 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:33.152+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:33.226 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.226 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:33.254 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:33.254+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:33.342 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.342 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:33.354 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:33.355+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.453 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.455 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:33.453+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:33.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:33.556 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:33.555+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.604 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:33.696 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:33.695+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:33.754 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.755 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.757 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:33.755+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:33.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:33.858 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:33.858+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:33.905 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:33.905 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:33.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:33.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:33.959 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:33.959+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:34.018 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:34.018 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:34.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:34.032 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:34.062 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:34.071 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:34.071+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:34.078 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:34.078 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:34.094 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:34.094 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:34.109 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:34.131 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:34.131 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:34.132 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:34.132 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:34.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:34.161 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:34.161+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:34.261 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:34.260+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:34.277 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:34.277 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:34.277 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:34.278 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:34.278 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:34.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:34.362 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:34.362+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:34.385 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:34.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.218684857, 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.386 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:34.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.21134552300001, 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:34.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.21062844900001, 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:34.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.20140088799999, 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:34.399 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:34.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:34.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.19217631999999, 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:34.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.17670168000001, 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.408 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:34.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.16741217000001, 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.16082192499999, 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.417 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:34.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:34.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:34.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.14463944400001, 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.422 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:34.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.13556452399999, 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:34.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:34.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.79512564800001, 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:34.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.436 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.79307337099999, 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:34.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.73284172000001, 2024-09-17T07:51:34.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:34.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 201.57166257099999, 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.448 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:34.453 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 201.57163826999999, 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.454 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:34.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 201.57160389399999, 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.459 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:34.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 201.57158253700001, 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:34.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 201.568913893, 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 201.55919632000001, 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:34.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:34.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:34.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 194.59308593099999, 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:34.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:34.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.488 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:34.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:34.490 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:34.489+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:34.587 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:34.588 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:34.588 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:34.588 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:34.590 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:34.589+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:34.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:34.642 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:34.642 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:34.642 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 4, 1, 5] 2024-09-17T07:51:34.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-09-17T07:51:34.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:51:34.694 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:34.694 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:34.694 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:34.694 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:34.695 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:34.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:34.730 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:34.728+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:34.792 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:34.791+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:34.803 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.133874567000007, 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:34.807 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.130861694000004, 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:34.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.811 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.128594769000003, 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:34.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:34.816 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.126425151000007, 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.820 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.123135783999999, 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:34.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:34.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.117629655000002, 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:34.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:34.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.114791370000006, 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:34.832 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.110647530999998, 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.836 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.096412674999996, 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:34.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.840 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.085958219000005, 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.844 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.058248848999995, 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.043708426999999, 2024-09-17T07:51:34.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:34.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.039975291999994, 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.852 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.856 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.028933609000006, 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:34.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.014558823000002, 2024-09-17T07:51:34.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.000602948999997, 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.867 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.996968866000003, 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:34.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:34.871 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.991963005999999, 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.924682844000003, 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.875 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:34.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.915758775999997, 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:34.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:34.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:34.882 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:34.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:34.927 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:34.926+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-17T07:51:34.936+0000 7ffb533f1640 -1 mon.a@0(leader).osd e153 definitely_dead 0 2024-09-17T07:51:34.985 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:34.986 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:34.986 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:34.986 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:34.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:34.995 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:34.993+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:35.094 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:35.094+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:35.137 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:35.138 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:35.146 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:35.196 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:35.196+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.443122236999997, 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.441629214000002, 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.436643631000003, 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.432504108000003, 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.427719787000001, 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:35.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.41994854, 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:35.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:35.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.409910299000003, 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.274 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:35.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.390592738999999, 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:35.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:35.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:35.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.282 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.378194430999997, 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.286 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.374542875000003, 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:35.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:35.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.373959724000002, 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:35.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:35.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.369376381999999, 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.298 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:35.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.369047330999997, 2024-09-17T07:51:35.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.306 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.360848134999998, 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:35.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.310 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 49.360326888000003, 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:35.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:35.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.315 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.730040080999999, 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:35.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.319 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.705690279999999, 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:35.323 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.704163471000001, 2024-09-17T07:51:35.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.672918465000002, 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 45.621095406000002, 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:35.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:35.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:35.337 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:35.339 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:35.339 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:35.338+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:35.439 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:35.439+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:35.445 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:35.446 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:35.446 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:35.446 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:35.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:35.541 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:35.541+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:35.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:35.642 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:35.641+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:51:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.5. 2024-09-17T07:51:35.703 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:35.703 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:35.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:35.703 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.185668933000002, 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:35.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:35.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.170049267000003, 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.165968444999997, 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:35.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:35.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.159478303, 2024-09-17T07:51:35.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:35.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:35.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.153763093000002, 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:35.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.146525457999999, 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:35.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:35.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.138994853, 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:35.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 46.135219618999997, 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:35.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.183852813000001, 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.179143955000001, 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:35.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.175246903000001, 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:35.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:35.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:35.747 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.171657727000003, 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:35.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.751 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.167173409999997, 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:35.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.163241380000002, 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:35.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.159454541000002, 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:35.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.155802858999998, 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.151694417999998, 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:35.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.772 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.148094346999997, 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:35.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:35.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:35.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.144727754000002, 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:35.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:35.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 45.141336019999997, 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:35.781 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:35.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:35.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:35.785 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:35.787 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:35.786+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:35.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:35.887 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:35.887+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:35.893 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:35.893 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:35.902 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:35.988 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:35.988+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:36.044 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:36.045 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:36.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:36.094 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:36.094+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:36.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:36.190 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:36.189+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:36.197 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.03111974999999, 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.023460234, 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.01507066600001, 2024-09-17T07:51:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.01469941600001, 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.00663645099999, 2024-09-17T07:51:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.00626533900001, 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.99698493400001, 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.995325501, 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.989300676, 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.989050349, 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.98193651099999, 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.98141941200001, 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.96244824499999, 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.95619138500001, 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.949655323, 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.576788913, 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:36.278 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.57482850599999, 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:36.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:36.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:36.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:36.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 207.68915987, 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:36.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:36.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:36.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.37831423200001, 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:36.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:36.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.37695278699999, 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:36.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:36.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:36.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:36.303 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:36.305 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:36.305 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:36.305 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:36.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:36.306 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:36.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:36.307 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:36.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:36.307 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:36.304+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:36.316 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:36.316 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:36.332 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:36.333 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:36.405 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:36.405+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:36.418 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:36.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:36.506 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:36.506+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:36.569 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:36.569 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:36.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:36.621 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:36.621+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:36.687 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:36.688 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:36.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:36.741 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:36.741+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:36.808 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:36.808+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:36.812 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:36.812 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:36.822 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:36.909 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:36.909+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:36.961 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:36.961 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:36.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:37.010 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:37.010+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:37.107 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.107 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.111 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:37.111+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:37.115 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:37.212 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:37.212+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:37.254 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.254 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:37.313 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:37.312+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:37.401 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.401 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:37.452 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:37.452+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:37.507 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.508 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.514 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:37.513+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:37.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:37.615 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:37.615+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:37.628 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.628 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:37.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:51:37.716 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:37.716+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:37.741 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.742 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:37.817 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:37.817+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:37.857 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.857 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:37.918 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:37.918+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:37.965 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:37.965 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:37.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:38.057 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:38.057+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:38.111 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:38.112 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:38.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:38.121 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:38.120+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:38.221 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:38.221+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:38.259 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:38.259 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:38.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:38.322 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:38.322+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:38.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:38.371 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:38.371 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:38.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:38.423 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:38.423+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:38.462 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:38.462 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:38.478 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:38.478 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:38.494 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:38.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:38.510 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:38.510 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:38.524 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:38.524+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:38.525 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:38.525 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:38.525 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:38.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:38.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:38.542 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:38.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:38.626 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:38.626+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:38.645 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:38.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.478907741, 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.646 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.47124822500001, 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.462858657, 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:38.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.462487407, 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.454424442, 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.666 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:38.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.45405332999999, 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.670 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:38.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:38.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.444772925, 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:38.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:38.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.44311349200001, 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:38.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:38.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.43708866700001, 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:38.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.43683834000001, 2024-09-17T07:51:38.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.429724502, 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:38.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.700 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.42920740299999, 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.410236236, 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:38.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.403979376, 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.39744331399999, 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:38.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:38.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.02457690400001, 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:38.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:38.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.022616497, 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:38.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:38.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.13694786100001, 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:38.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:38.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 205.82610222299999, 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 205.82474077800001, 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:38.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:38.750 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:38.751 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:51:38.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:38.752 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:38.751+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:38.852 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:38.852+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:38.894 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:38.894 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:38.894 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:38.894 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:38.902 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:38.988 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:38.988+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:39.043 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:39.043 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:39.043 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:39.044 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:39.044 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:39.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:39.093 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:39.093+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:39.154 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:39.154+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:39.195 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.936877826, 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:39.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:39.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:39.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:39.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.56857221000001, 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:39.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.87638995500001, 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:39.209 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.870533099, 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.85884014000001, 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.215 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:39.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:39.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:39.220 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.78998555699999, 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:39.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:39.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.77808626800001, 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:39.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.72724172100001, 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.72123351400001, 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:39.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.71553315700001, 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.243 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:39.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:39.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 210.70979353199999, 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.249 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:39.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:39.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.254 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 206.37668248400001, 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:39.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:39.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.53158158700001, 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:39.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:39.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.525494864, 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:39.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:39.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:39.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.44203856300001, 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:39.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:39.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.435724131, 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.278 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:39.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.429352671, 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:39.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.42343019500001, 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:39.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:39.295 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 203.36544729900001, 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 199.407742627, 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:39.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:39.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:39.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:39.305 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:39.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:39.307 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:39.306+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:39.407 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:39.407+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:39.415 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:39.415 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:39.415 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:39.415 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:39.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:39.508 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:39.508+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:39.562 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:39.562 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:39.562 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:39.562 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:39.562 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:39.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:39.614 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:39.614+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:39.671 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.505203956, 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.672 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:39.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.49786462200001, 2024-09-17T07:51:39.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:39.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.49714754799999, 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:39.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:39.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.685 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.487919987, 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:39.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.47869541899999, 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.46322077900001, 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.698 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.45393126900001, 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:39.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.447341024, 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:39.706 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.43115854300001, 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.707 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:39.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:39.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.42208362299999, 2024-09-17T07:51:39.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:39.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.715 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.08164474700001, 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.716 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:39.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.721 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.07959246999999, 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:39.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:39.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:39.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.01936081900001, 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:39.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:39.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:39.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:39.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.85818166999999, 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:39.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.858157369, 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:39.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.85812299299999, 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:39.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.85810163599999, 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.855432992, 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.84571541899999, 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:39.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.87960502999999, 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:39.764 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:39.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:39.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:39.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.769 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:39.770 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:39.770 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:39.771 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:39.772 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:39.771+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:39.872 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:39.871+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:39.910 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:39.910 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:39.910 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:39.910 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:39.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:39.972 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:39.972+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:40.016 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:40.016 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:40.016 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:40.016 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:40.016 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:40.024 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:40.108 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:40.108+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:40.163 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.494161262000006, 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:40.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.491148389000003, 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:40.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.488881464000002, 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:40.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:40.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.486711846000006, 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.483422478999998, 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.477916350000001, 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.475078065000005, 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.470934225999997, 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.456699369999995, 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.446244914000005, 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.418535543999994, 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:40.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.403995121999998, 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:40.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:40.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.400261986999993, 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.214 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.389220304000006, 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:40.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.218 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.374845518000001, 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.360889643999997, 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:40.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.357255561000002, 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:40.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:40.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:40.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.352249700999998, 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.284969539000002, 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.276045471000003, 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:40.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:40.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:40.242 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:40.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:40.245 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:40.243+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:40.345 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:40.345+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:40.359 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:40.360 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:40.360 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:40.360 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:40.369 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:40.446 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:40.446+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:40.471 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:40.472 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:40.472 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:40.472 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:40.472 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:40.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:40.547 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:40.547+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:40.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:40.585 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.778382579000002, 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.776889556, 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.771903973000001, 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.767764450000001, 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.762980128999999, 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.755208881999998, 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.745170641000001, 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.725853080999997, 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.713454773000002, 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.709803217000001, 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.709220066, 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.704636723999997, 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.704307673000002, 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.696108477000003, 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.695587230000001, 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.065300422999996, 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.040950621999997, 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.039423812999999, 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 51.008178807, 2024-09-17T07:51:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 50.956355748, 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:40.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:40.685 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:40.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:40.686 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:40.686 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-17T07:51:40.686 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:51:40.686 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:51:40.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-17T07:51:40.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:40.688 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:40.689 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:40.689 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:40.686+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:40.690 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:40.690 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:40.704 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:40.721 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:40.737 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:40.737 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:40.752 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:40.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:51:40.787 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:40.787+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:40.792 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-09-17T07:51:40.793 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:40.794 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:40.794 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:40.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:40.888 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:40.888+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:40.940 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:40.940 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:40.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:40.992 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:40.992+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:41.045 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.046 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:41.078 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd1 2024-09-17T07:51:41.078 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:41.078 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:41.078 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:41.078 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:41.079 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd1'\'']/id' 2024-09-17T07:51:41.098 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:41.098+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:41.153 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30408 2024-09-17T07:51:41.153 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30408 2024-09-17T07:51:41.153 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30408 -C rbd-nbd 2024-09-17T07:51:41.157 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:41.158 INFO:tasks.workunit.client.0.smithi098.stdout: 30408 ? 00:00:00 rbd-nbd 2024-09-17T07:51:41.158 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-17T07:51:41.158 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:41.158 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:41.159 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:41.159 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:41.159 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.160 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.161 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:41.161 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:41.161 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-17T07:51:41.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:41.170 INFO:tasks.workunit.client.0.smithi098.stderr:blkdiscard: /dev/nbd1 contains existing file system (ext2). 2024-09-17T07:51:41.172 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-17T07:51:41.172 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:41.172 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:41.172 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:41.173 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:41.175 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:41.175 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:41.175 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-17T07:51:41.184 INFO:tasks.workunit.client.0.smithi098.stderr:blkdiscard: /dev/nbd1 contains existing file system (ext2). 2024-09-17T07:51:41.189 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-17T07:51:41.189 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:41.189 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:41.190 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:41.190 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:41.190+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:41.190 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:41.192 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:41.192 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:41.192 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-17T07:51:41.207 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-17T07:51:41.207 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:41.207 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:41.208 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:41.208 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:41.210 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:41.210 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:41.210 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-17T07:51:41.225 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-17T07:51:41.225 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:41.226 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:41.227 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:41.227 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:41.228 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:41.228 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:41.228 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-17T07:51:41.243 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-17T07:51:41.243 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:41.243 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:41.244 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:41.244 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:41.246 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:41.246 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:41.246 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-17T07:51:41.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ tries=0 2024-09-17T07:51:41.263 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27356 2024-09-17T07:51:41.263 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-17T07:51:41.291 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:41.291+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:41.315 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:41.314+0000 7fcdc9949640 -1 JournalMetadata: 0x55cb75a23890 failed to locate client: INSPECT 2024-09-17T07:51:41.318 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.318 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.318 INFO:tasks.workunit.client.0.smithi098.stderr:+ queue_length=4 2024-09-17T07:51:41.318 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -lt 10 ']' 2024-09-17T07:51:41.318 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 4 -gt 0 ']' 2024-09-17T07:51:41.318 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27356 --verbose 2024-09-17T07:51:41.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:41.365 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=1 2024-09-17T07:51:41.366 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:41.366 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:41.366 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:41.366 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.185146+0000" 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=2 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.203173+0000" 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=3 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:41.367 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.238706+0000" 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=4 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:41.368 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.258689+0000" 2024-09-17T07:51:41.369 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:41.369 INFO:tasks.workunit.client.0.smithi098.stdout:Summary: 2024-09-17T07:51:41.369 INFO:tasks.workunit.client.0.smithi098.stdout: 4 entries inspected, 0 errors 2024-09-17T07:51:41.369 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:41.368+0000 7f97fd445640 -1 JournalMetadata: 0x55b044bfa370 failed to locate client: INSPECT 2024-09-17T07:51:41.371 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 1 2024-09-17T07:51:41.391 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:41.391+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:41.430 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.430 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:41.492 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:41.492+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:41.581 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.581 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:41.592 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:41.592+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:41.662 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:51:41.693 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.693 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.694 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:41.694+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:41.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:41.794 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:41.794+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:41.845 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.845 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:41.897 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:41.897+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:41.952 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:41.953 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:41.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:41.997 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:41.997+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:42.098 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:42.098+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:42.099 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.099 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:42.199 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:42.199+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:42.204 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.204 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.213 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:42.300 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:42.300+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:42.352 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.352 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:42.373 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27356 2024-09-17T07:51:42.373 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-17T07:51:42.404 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:42.404+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:42.434 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:42.434+0000 7fc4b27fc640 -1 JournalMetadata: 0x559f210b1e70 failed to locate client: INSPECT 2024-09-17T07:51:42.439 INFO:tasks.workunit.client.0.smithi098.stderr:+ queue_length=4 2024-09-17T07:51:42.439 INFO:tasks.workunit.client.0.smithi098.stderr:+ tries=1 2024-09-17T07:51:42.439 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1 -lt 10 ']' 2024-09-17T07:51:42.439 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 4 -gt 0 ']' 2024-09-17T07:51:42.439 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27356 --verbose 2024-09-17T07:51:42.461 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.461 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=1 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.185146+0000" 2024-09-17T07:51:42.491 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=2 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.203173+0000" 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=3 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:42.492 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.238706+0000" 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=4 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.258689+0000" 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout:Summary: 2024-09-17T07:51:42.493 INFO:tasks.workunit.client.0.smithi098.stdout: 4 entries inspected, 0 errors 2024-09-17T07:51:42.494 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:42.493+0000 7f2318d9c640 -1 JournalMetadata: 0x55e2d0779cd0 failed to locate client: INSPECT 2024-09-17T07:51:42.497 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 1 2024-09-17T07:51:42.511 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:42.511+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:42.602 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:42.602+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:42.608 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.608 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.617 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:42.703 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:42.703+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:42.727 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.727 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:42.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:42.822 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:42.822+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:42.838 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:42.838 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:42.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:42.896 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:42.897 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:42.904 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:42.904+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:42.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:42.930 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:42.930 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:42.946 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:42.946 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:42.961 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:42.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:42.977 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:42.995 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:42.995 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.189514549999998, 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:42.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:42.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:42.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:42.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.188021526999997, 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:43.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.183035943999997, 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.178896420999997, 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:43.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:43.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.174112100000002, 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:43.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.166340853000001, 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:43.019 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.156302611999998, 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.136985052, 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.023 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:43.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.124586743999998, 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.027 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:43.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.120935187999997, 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:43.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.120352037000004, 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.035 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:43.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.115768695, 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.039 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:43.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.115439643999999, 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:43.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.107240447999999, 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:43.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:43.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 57.106719200999997, 2024-09-17T07:51:43.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.476432394, 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:43.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:43.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.452082593, 2024-09-17T07:51:43.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:43.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:43.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.450555784000002, 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:43.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:43.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:43.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.419310778000003, 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:43.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:43.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 53.367487719000003, 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:43.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:43.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:43.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.078 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:43.079 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:43.079 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:43.078+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:43.177 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:43.178 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:43.178 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:43.178 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:43.180 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:43.179+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:43.186 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:43.280 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:43.280+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:43.324 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:43.324 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:43.324 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:43.332 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:43.382 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:43.381+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:43.431 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.431 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.913486851999998, 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:43.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.897867185999999, 2024-09-17T07:51:43.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:43.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:43.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.893786364, 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:43.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:43.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.887296222000003, 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:43.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.881581011999998, 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:43.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.874343377000002, 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:43.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.866812772000003, 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.454 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.458 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.863037538, 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:43.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.911670731999997, 2024-09-17T07:51:43.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:43.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.906961873999997, 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:43.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:43.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:43.470 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.903064821999997, 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:43.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:43.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.899475645999999, 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:43.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:43.478 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.894991329, 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:43.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.891059298999998, 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.887272459999998, 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.883620778000001, 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:43.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.879512337000001, 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:43.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:43.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.875912266, 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.500 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.872545672999998, 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:43.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.508 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 52.869153939, 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:43.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:43.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:43.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:43.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:43.513 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:43.514 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27356 2024-09-17T07:51:43.514 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-17T07:51:43.515 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:43.513+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:43.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:43.549 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:43.549+0000 7faa9659f640 -1 JournalMetadata: 0x55be15dc80f0 failed to locate client: INSPECT 2024-09-17T07:51:43.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ queue_length=4 2024-09-17T07:51:43.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ tries=2 2024-09-17T07:51:43.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 2 -lt 10 ']' 2024-09-17T07:51:43.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 4 -gt 0 ']' 2024-09-17T07:51:43.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27356 --verbose 2024-09-17T07:51:43.603 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=1 2024-09-17T07:51:43.603 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:43.603 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:43.603 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:43.603 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:43.603 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.185146+0000" 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=2 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.203173+0000" 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=3 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:43.604 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.238706+0000" 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=4 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.258689+0000" 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout:Summary: 2024-09-17T07:51:43.605 INFO:tasks.workunit.client.0.smithi098.stdout: 4 entries inspected, 0 errors 2024-09-17T07:51:43.606 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:43.606+0000 7ff0ddffb640 -1 JournalMetadata: 0x557b86f90900 failed to locate client: INSPECT 2024-09-17T07:51:43.610 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 1 2024-09-17T07:51:43.615 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:43.615+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:43.628 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:43.628 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:43.628 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:43.628 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:43.638 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:43.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:51:43.722 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:43.722+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:43.781 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:43.782 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:43.782 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:43.782 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:43.782 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:43.791 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:43.817 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:43.817+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:43.919 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:43.919+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:43.934 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:43.934 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.76814381200001, 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:43.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:43.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.76048429599999, 2024-09-17T07:51:43.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:43.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:43.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.944 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.752094728, 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:43.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:43.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:43.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:43.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.751723478, 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:43.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:43.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.74366051300001, 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:43.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:43.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:43.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.743289401, 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:43.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.734008996, 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:43.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:43.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.73234956300001, 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:43.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.72632473799999, 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.974 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:43.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.72607441100001, 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:43.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.718960573, 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:43.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:43.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.718443474, 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:43.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.993 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.69947230700001, 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:43.995 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:43.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:43.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:43.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:43.999 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.693215447, 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.003 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.68667938499999, 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:44.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:44.008 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.31381297499999, 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.31185256800001, 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:44.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:44.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.42618393199999, 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:44.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:44.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:44.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:44.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.115338294, 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:44.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:44.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:44.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.11397684900001, 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.030 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:44.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:44.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.036 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:44.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:44.038 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:44.036+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:44.138 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:44.137+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:44.180 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:44.181 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:44.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:44.181 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:44.189 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:44.239 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:44.238+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:44.290 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:44.290 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:44.290 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:44.290 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:44.290 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:44.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:44.342 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:44.342+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:44.402 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 222.143302651, 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.403 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.404 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.405 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.77499703500001, 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:44.407 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.408 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.409 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.411 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.08281478000001, 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.412 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.076957924, 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.06526496500001, 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.99641038199999, 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:44.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.98451109300001, 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.435 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.436 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:44.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:44.439 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:44.440 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.93366654600001, 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.441 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:44.443 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.444 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:44.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:44.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.927658339, 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:44.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.448 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:44.450 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:44.451 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.92195798200001, 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:44.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:44.456 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.457 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 215.91621835699999, 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.459 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:44.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:44.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:44.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:44.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.464 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.58310730900001, 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:44.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.738006412, 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:44.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.475 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:44.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.73191968899999, 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.477 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:44.478 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.481 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:44.482 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.64846338800001, 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:44.484 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.485 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.486 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:44.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.642148956, 2024-09-17T07:51:44.488 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.490 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.491 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.492 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:44.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.635777496, 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:44.494 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.495 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:44.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.62985502000001, 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:44.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:44.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 208.57187212400001, 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:44.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 204.614167452, 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:44.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:44.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:44.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:44.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:44.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:44.516 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:44.517 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:44.516+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:44.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:44.612 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27356 2024-09-17T07:51:44.612 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-17T07:51:44.617 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:44.617+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:44.627 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:44.627 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:44.628 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:44.628 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:44.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:44.670 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:51:44.670 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:44.670+0000 7f726a847640 -1 JournalMetadata: 0x558b4d0fa160 failed to locate client: INSPECT 2024-09-17T07:51:44.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ queue_length=4 2024-09-17T07:51:44.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ tries=3 2024-09-17T07:51:44.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 3 -lt 10 ']' 2024-09-17T07:51:44.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 4 -gt 0 ']' 2024-09-17T07:51:44.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27356 --verbose 2024-09-17T07:51:44.718 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:44.717+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:44.727 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=1 2024-09-17T07:51:44.727 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:44.727 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:44.727 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:44.727 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.185146+0000" 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=2 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.203173+0000" 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:44.728 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=3 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.238706+0000" 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=4 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.258689+0000" 2024-09-17T07:51:44.729 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:44.730 INFO:tasks.workunit.client.0.smithi098.stdout:Summary: 2024-09-17T07:51:44.730 INFO:tasks.workunit.client.0.smithi098.stdout: 4 entries inspected, 0 errors 2024-09-17T07:51:44.730 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:44.730+0000 7ff7c919e640 -1 JournalMetadata: 0x55ad6f328360 failed to locate client: INSPECT 2024-09-17T07:51:44.734 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 1 2024-09-17T07:51:44.737 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:44.738 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:44.738 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:44.738 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:44.738 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:44.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:44.819 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:44.818+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.71926876200001, 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:44.886 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.71192942799999, 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.895 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.711212354, 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:44.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.900 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.70198479300001, 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:44.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.692760225, 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.906 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.67728558499999, 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:44.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:44.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.66799607499999, 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.916 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:44.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.66140583000001, 2024-09-17T07:51:44.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.924 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.64522334899999, 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.925 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:44.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.929 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 222.636148429, 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:44.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.934 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.29570955299999, 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:44.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:44.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.940 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.293657276, 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.941 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:44.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.233425625, 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.947 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:44.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:44.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.072246476, 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:44.953 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.07222217500001, 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.959 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:44.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:44.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.964 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.07218779900001, 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:44.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:44.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.072166442, 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:44.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:44.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.06949779799999, 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:44.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:44.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.981 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.059780225, 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:44.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 205.093669836, 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:44.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:44.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:44.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:44.993 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:44.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:44.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:44.996 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:44.993+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:45.095 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:45.094+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:45.111 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:45.111 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:45.127 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:45.127 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:45.134 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:45.134 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:45.134 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:45.135 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:45.142 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:45.142 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:45.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:45.158 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:45.174 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:45.174 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:45.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:45.189 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:45.227 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:45.226+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:45.281 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:45.281 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:45.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:45.331 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:45.330+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:45.398 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:45.397+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:45.427 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:45.427 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:45.436 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:45.498 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:45.498+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:45.539 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:45.539 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:45.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:45.599 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:45.598+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:45.691 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:45.692 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:45.700 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:45.701 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:45.699+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:45.736 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27356 2024-09-17T07:51:45.790 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-17T07:51:45.793 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:45.792+0000 7f58f1247640 -1 JournalMetadata: 0x5577ade14fa0 failed to locate client: INSPECT 2024-09-17T07:51:45.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ queue_length=4 2024-09-17T07:51:45.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ tries=4 2024-09-17T07:51:45.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 4 -lt 10 ']' 2024-09-17T07:51:45.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 4 -gt 0 ']' 2024-09-17T07:51:45.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27356 --verbose 2024-09-17T07:51:45.800 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:45.800+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:45.802 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:45.802 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:45.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:45.804 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:45.804 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:45.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:45.846 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=1 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.185146+0000" 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=2 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 0, 2024-09-17T07:51:45.847 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.203173+0000" 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=3 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.238706+0000" 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:45.848 INFO:tasks.workunit.client.0.smithi098.stdout:Entry: tag_id=1, commit_tid=4 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout:{ 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout: "event_type": "AioDiscard", 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout: "offset": 4096, 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout: "length": 4096, 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout: "discard_granularity_bytes": 4096, 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout: "timestamp": "2024-09-17T07:51:41.258689+0000" 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout:} 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout:Summary: 2024-09-17T07:51:45.849 INFO:tasks.workunit.client.0.smithi098.stdout: 4 entries inspected, 0 errors 2024-09-17T07:51:45.850 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:45.849+0000 7f9b617fa640 -1 JournalMetadata: 0x561afab9ed80 failed to locate client: INSPECT 2024-09-17T07:51:45.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 1 2024-09-17T07:51:45.901 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:45.900+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:45.912 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:45.912 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:45.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:46.002 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:46.002+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:46.018 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.018 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:46.074 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:46.075 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:46.086 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-17T07:51:45.467969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229749047,"num_objects":132,"num_object_clones":19,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":12,"num_read":6367,"num_read_kb":192844,"num_write":944,"num_write_kb":274759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":364972256,"num_keys_recovered":63,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":48,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1961816,"kb_used_data":448736,"kb_used_omap":15,"kb_used_meta":1513008,"kb_avail":469897384,"statfs":{"total":483183820800,"available":481174921216,"internally_reserved":0,"allocated":459505664,"data_stored":464873503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15949,"internal_metadata":1549320627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6768,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-13,"num_objects_misplaced":-8,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":450,"num_read_kb":4734,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":42009792,"num_keys_recovered":3,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.816383"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.652398+0000","last_change":"2024-09-17T07:51:36.651435+0000","last_active":"2024-09-17T07:51:36.652398+0000","last_peered":"2024-09-17T07:51:36.652398+0000","last_clean":"2024-09-17T07:51:36.652398+0000","last_became_active":"2024-09-17T07:51:36.651071+0000","last_became_peered":"2024-09-17T07:51:36.651071+0000","last_unstale":"2024-09-17T07:51:36.652398+0000","last_undegraded":"2024-09-17T07:51:36.652398+0000","last_fullsized":"2024-09-17T07:51:36.652398+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:10:40.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000513523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"156'10","reported_seq":76,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:41.187188+0000","last_change":"2024-09-17T07:50:49.578552+0000","last_active":"2024-09-17T07:51:41.187188+0000","last_peered":"2024-09-17T07:51:41.187188+0000","last_clean":"2024-09-17T07:51:41.187188+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:41.187188+0000","last_undegraded":"2024-09-17T07:51:41.187188+0000","last_fullsized":"2024-09-17T07:51:41.187188+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028777799999999998,"stat_sum":{"num_bytes":16777370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":9,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"138'52","reported_seq":629,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:41.300044+0000","last_change":"2024-09-17T07:51:36.877931+0000","last_active":"2024-09-17T07:51:41.300044+0000","last_peered":"2024-09-17T07:51:41.300044+0000","last_clean":"2024-09-17T07:51:41.300044+0000","last_became_active":"2024-09-17T07:51:36.653275+0000","last_became_peered":"2024-09-17T07:51:36.653275+0000","last_unstale":"2024-09-17T07:51:41.300044+0000","last_undegraded":"2024-09-17T07:51:41.300044+0000","last_fullsized":"2024-09-17T07:51:41.300044+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.123023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16778394,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":1236,"num_read_kb":34054,"num_write":40,"num_write_kb":16451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":37815682,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"57'1","reported_seq":19,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.647597+0000","last_change":"2024-09-17T07:51:30.634526+0000","last_active":"2024-09-17T07:51:36.647597+0000","last_peered":"2024-09-17T07:51:36.647597+0000","last_clean":"2024-09-17T07:51:36.647597+0000","last_became_active":"2024-09-17T07:51:30.632129+0000","last_became_peered":"2024-09-17T07:51:30.632129+0000","last_unstale":"2024-09-17T07:51:36.647597+0000","last_undegraded":"2024-09-17T07:51:36.647597+0000","last_fullsized":"2024-09-17T07:51:36.647597+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:06:10.245955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071767599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"54'4","reported_seq":112,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.653034+0000","last_change":"2024-09-17T07:51:36.679420+0000","last_active":"2024-09-17T07:51:38.653034+0000","last_peered":"2024-09-17T07:51:38.653034+0000","last_clean":"2024-09-17T07:51:38.653034+0000","last_became_active":"2024-09-17T07:51:36.648087+0000","last_became_peered":"2024-09-17T07:51:36.648087+0000","last_unstale":"2024-09-17T07:51:38.653034+0000","last_undegraded":"2024-09-17T07:51:38.653034+0000","last_fullsized":"2024-09-17T07:51:38.653034+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:03.506247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031538300000000003,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"156'412","reported_seq":1937,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:41.365619+0000","last_change":"2024-09-17T07:50:47.509256+0000","last_active":"2024-09-17T07:51:41.365619+0000","last_peered":"2024-09-17T07:51:41.365619+0000","last_clean":"2024-09-17T07:51:41.365619+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:41.365619+0000","last_undegraded":"2024-09-17T07:51:41.365619+0000","last_fullsized":"2024-09-17T07:51:41.365619+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:16.833359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692729,"num_objects":28,"num_object_clones":10,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":2128,"num_read_kb":86809,"num_write":495,"num_write_kb":66616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":17,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":157,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.651652+0000","last_change":"2024-09-17T07:51:36.650992+0000","last_active":"2024-09-17T07:51:36.651652+0000","last_peered":"2024-09-17T07:51:36.651652+0000","last_clean":"2024-09-17T07:51:36.651652+0000","last_became_active":"2024-09-17T07:51:36.650696+0000","last_became_peered":"2024-09-17T07:51:36.650696+0000","last_unstale":"2024-09-17T07:51:36.651652+0000","last_undegraded":"2024-09-17T07:51:36.651652+0000","last_fullsized":"2024-09-17T07:51:36.651652+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:03:48.887361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091099400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"138'6","reported_seq":45,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.647729+0000","last_change":"2024-09-17T07:50:53.576654+0000","last_active":"2024-09-17T07:51:36.647729+0000","last_peered":"2024-09-17T07:51:36.647729+0000","last_clean":"2024-09-17T07:51:36.647729+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:36.647729+0000","last_undegraded":"2024-09-17T07:51:36.647729+0000","last_fullsized":"2024-09-17T07:51:36.647729+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078848299999999998,"stat_sum":{"num_bytes":12582969,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":6,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"156'286","reported_seq":1504,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:41.369432+0000","last_change":"2024-09-17T07:50:17.507640+0000","last_active":"2024-09-17T07:51:41.369432+0000","last_peered":"2024-09-17T07:51:41.369432+0000","last_clean":"2024-09-17T07:51:41.369432+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:41.369432+0000","last_undegraded":"2024-09-17T07:51:41.369432+0000","last_fullsized":"2024-09-17T07:51:41.369432+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18095537,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":2597,"num_read_kb":47168,"num_write":146,"num_write_kb":33830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":42,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.652805+0000","last_change":"2024-09-17T07:50:45.802304+0000","last_active":"2024-09-17T07:51:38.652805+0000","last_peered":"2024-09-17T07:51:38.652805+0000","last_clean":"2024-09-17T07:51:38.652805+0000","last_became_active":"2024-09-17T07:50:45.801997+0000","last_became_peered":"2024-09-17T07:50:45.801997+0000","last_unstale":"2024-09-17T07:51:38.652805+0000","last_undegraded":"2024-09-17T07:51:38.652805+0000","last_fullsized":"2024-09-17T07:51:38.652805+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"156'13","reported_seq":276,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:41.240798+0000","last_change":"2024-09-17T07:51:36.828061+0000","last_active":"2024-09-17T07:51:41.240798+0000","last_peered":"2024-09-17T07:51:41.240798+0000","last_clean":"2024-09-17T07:51:41.240798+0000","last_became_active":"2024-09-17T07:51:36.652299+0000","last_became_peered":"2024-09-17T07:51:36.652299+0000","last_unstale":"2024-09-17T07:51:41.240798+0000","last_undegraded":"2024-09-17T07:51:41.240798+0000","last_fullsized":"2024-09-17T07:51:41.240798+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:34:56.626824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054065899999999997,"stat_sum":{"num_bytes":37748930,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":75497548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":215,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.649471+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:36.649471+0000","last_peered":"2024-09-17T07:51:36.649471+0000","last_clean":"2024-09-17T07:51:36.649471+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:36.649471+0000","last_undegraded":"2024-09-17T07:51:36.649471+0000","last_fullsized":"2024-09-17T07:51:36.649471+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"137'91","reported_seq":337,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.652735+0000","last_change":"2024-09-17T07:50:50.567734+0000","last_active":"2024-09-17T07:51:38.652735+0000","last_peered":"2024-09-17T07:51:38.652735+0000","last_clean":"2024-09-17T07:51:38.652735+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:38.652735+0000","last_undegraded":"2024-09-17T07:51:38.652735+0000","last_fullsized":"2024-09-17T07:51:38.652735+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373608,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":222,"num_read_kb":24678,"num_write":98,"num_write_kb":25130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"57'1","reported_seq":49,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.700476+0000","last_change":"2024-09-17T07:51:36.700036+0000","last_active":"2024-09-17T07:51:36.700476+0000","last_peered":"2024-09-17T07:51:36.700476+0000","last_clean":"2024-09-17T07:51:36.700476+0000","last_became_active":"2024-09-17T07:51:36.651212+0000","last_became_peered":"2024-09-17T07:51:36.651212+0000","last_unstale":"2024-09-17T07:51:36.700476+0000","last_undegraded":"2024-09-17T07:51:36.700476+0000","last_fullsized":"2024-09-17T07:51:36.700476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:28.934409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"156'8","reported_seq":257,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:41.260383+0000","last_change":"2024-09-17T07:50:52.613371+0000","last_active":"2024-09-17T07:51:41.260383+0000","last_peered":"2024-09-17T07:51:41.260383+0000","last_clean":"2024-09-17T07:51:41.260383+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:41.260383+0000","last_undegraded":"2024-09-17T07:51:41.260383+0000","last_fullsized":"2024-09-17T07:51:41.260383+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036208300000000002,"stat_sum":{"num_bytes":16777294,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":14,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"57'3","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.652837+0000","last_change":"2024-09-17T07:51:36.681335+0000","last_active":"2024-09-17T07:51:38.652837+0000","last_peered":"2024-09-17T07:51:38.652837+0000","last_clean":"2024-09-17T07:51:38.652837+0000","last_became_active":"2024-09-17T07:51:36.649783+0000","last_became_peered":"2024-09-17T07:51:36.649783+0000","last_unstale":"2024-09-17T07:51:38.652837+0000","last_undegraded":"2024-09-17T07:51:38.652837+0000","last_fullsized":"2024-09-17T07:51:38.652837+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:38:53.561331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032520599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"45'5","reported_seq":166,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.649668+0000","last_change":"2024-09-17T07:50:45.870519+0000","last_active":"2024-09-17T07:51:36.649668+0000","last_peered":"2024-09-17T07:51:36.649668+0000","last_clean":"2024-09-17T07:51:36.649668+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:36.649668+0000","last_undegraded":"2024-09-17T07:51:36.649668+0000","last_fullsized":"2024-09-17T07:51:36.649668+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"57'2","reported_seq":63,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.652913+0000","last_change":"2024-09-17T07:51:36.684058+0000","last_active":"2024-09-17T07:51:38.652913+0000","last_peered":"2024-09-17T07:51:38.652913+0000","last_clean":"2024-09-17T07:51:38.652913+0000","last_became_active":"2024-09-17T07:51:36.650971+0000","last_became_peered":"2024-09-17T07:51:36.650971+0000","last_unstale":"2024-09-17T07:51:38.652913+0000","last_undegraded":"2024-09-17T07:51:38.652913+0000","last_fullsized":"2024-09-17T07:51:38.652913+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:27.625741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371488,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":48,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"45'3","reported_seq":67,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.652888+0000","last_change":"2024-09-17T07:50:52.556404+0000","last_active":"2024-09-17T07:51:38.652888+0000","last_peered":"2024-09-17T07:51:38.652888+0000","last_clean":"2024-09-17T07:51:38.652888+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:38.652888+0000","last_undegraded":"2024-09-17T07:51:38.652888+0000","last_fullsized":"2024-09-17T07:51:38.652888+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"49'2","reported_seq":180,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T07:51:36.654401+0000","last_change":"2024-09-17T07:51:36.654078+0000","last_active":"2024-09-17T07:51:36.654401+0000","last_peered":"2024-09-17T07:51:36.654401+0000","last_clean":"2024-09-17T07:51:36.654401+0000","last_became_active":"2024-09-17T07:51:36.650903+0000","last_became_peered":"2024-09-17T07:51:36.650903+0000","last_unstale":"2024-09-17T07:51:36.654401+0000","last_undegraded":"2024-09-17T07:51:36.654401+0000","last_fullsized":"2024-09-17T07:51:36.654401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:48:02.630800+0000","last_clean_scrub_stamp":"2024-09-17T07:48:02.630800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:15.277183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070990300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":167,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-17T07:51:38.652732+0000","last_change":"2024-09-17T07:51:36.698276+0000","last_active":"2024-09-17T07:51:38.652732+0000","last_peered":"2024-09-17T07:51:38.652732+0000","last_clean":"2024-09-17T07:51:38.652732+0000","last_became_active":"2024-09-17T07:51:36.647444+0000","last_became_peered":"2024-09-17T07:51:36.647444+0000","last_unstale":"2024-09-17T07:51:38.652732+0000","last_undegraded":"2024-09-17T07:51:38.652732+0000","last_fullsized":"2024-09-17T07:51:38.652732+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:09:00.276726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034083200000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25165932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":137,"num_keys_recovered":26,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":94940268,"num_objects":71,"num_object_clones":19,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":6183,"num_read_kb":192709,"num_write":779,"num_write_kb":142027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":104924973,"num_keys_recovered":9,"num_objects_omap":6,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":48,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192995328,"data_stored":203789682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":841,"ondisk_log_size":841,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218341,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":58,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":260047146,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327282688,"data_stored":327157106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":0,"seq":412316860439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363696,"kb_used_data":58020,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94008144,"statfs":{"total":96636764160,"available":96264339456,"internally_reserved":0,"allocated":59412480,"data_stored":65377809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.32300000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.35899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.25900000000000001}]}]},{"osd":3,"up_from":126,"seq":541165879308,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415700,"kb_used_data":117768,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":93956140,"statfs":{"total":96636764160,"available":96211087360,"internally_reserved":0,"allocated":120594432,"data_stored":119455588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640317,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358268,"kb_used_data":52596,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94013572,"statfs":{"total":96636764160,"available":96269897728,"internally_reserved":0,"allocated":53858304,"data_stored":52744096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.34999999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.62}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.63300000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640318,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411656,"kb_used_data":105980,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93960184,"statfs":{"total":96636764160,"available":96215228416,"internally_reserved":0,"allocated":108523520,"data_stored":107394659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.54500000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56100000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.746},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.69499999999999995}]}]},{"osd":0,"up_from":121,"seq":519691042829,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412496,"kb_used_data":114372,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93959344,"statfs":{"total":96636764160,"available":96214368256,"internally_reserved":0,"allocated":117116928,"data_stored":119901351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67137536,"data_stored":67109151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75513856,"data_stored":75497744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92311552,"data_stored":92275168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48082944,"data_stored":51967353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30527488,"data_stored":30487115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30990336,"data_stored":30953049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25792512,"data_stored":25765186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44937216,"data_stored":51967353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1680,"internal_metadata":0}]}} 2024-09-17T07:51:46.103 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:46.102+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:46.124 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.124 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:46.204 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:46.204+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:46.235 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.235 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:46.328 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:46.328+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:46.386 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.386 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:46.405 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:46.404+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:46.506 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:46.504+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:46.535 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.536 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:46.606 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:46.606+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:46.647 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.647 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:46.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T07:51:46.740 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:46.740+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:46.800 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.800 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.808 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:46.808+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:46.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:46.854 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27356 2024-09-17T07:51:46.854 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-17T07:51:46.909 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:46.907+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:46.914 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:46.914+0000 7f3d6e94a640 -1 JournalMetadata: 0x55b9ebcca6a0 failed to locate client: INSPECT 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ queue_length=0 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ tries=5 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 5 -lt 10 ']' 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -gt 0 ']' 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd1 30408 2024-09-17T07:51:46.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd1 2024-09-17T07:51:46.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=30408 2024-09-17T07:51:46.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:46.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:46.920 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:46.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:46.920 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:46.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:46.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:46.922 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:46.955 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:46.955 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:46.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:47.009 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:47.008+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:47.015 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:47.015 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^30408\b' 2024-09-17T07:51:47.015 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^30408\b' 2024-09-17T07:51:47.063 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:47.063 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:47.063 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *30408\b' 2024-09-17T07:51:47.063 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *30408\b' 2024-09-17T07:51:47.068 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:47.068 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:47.073 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:47.073 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:47.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:47.110 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:47.110+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:47.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:47.211 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:47.209+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:47.222 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:47.222 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:47.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:47.315 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:47.314+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:47.367 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:47.368 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:47.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:47.417 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:47.416+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:47.513 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:47.513+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:47.515 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:47.515 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:47.520 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:47.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:47.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:47.536 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:47.536 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:47.551 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:47.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:47.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:47.566 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:47.569 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:51:47.569 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd feature disable rbd/testrbdnbd27356 journaling 2024-09-17T07:51:47.581 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:47.581 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:47.597 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:47.597 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:47.624 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:47.624 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:47.624 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:47.625 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:47.625 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:47.627 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd config image rm rbd/testrbdnbd27356 rbd_discard_granularity_bytes 2024-09-17T07:51:47.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:47.648 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:47.648+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:47.676 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-09-17T07:51:47.684 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:51:47.684 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:47.684 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:47.685 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:47.685 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:47.687 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:47.687 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:47.687 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:51:47.730 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:47.730+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:47.734 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:47.734 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:47.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:47.734 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:47.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.734 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.064800511, 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:47.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.061787638, 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:47.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.059520713, 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:47.748 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.057351095, 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:47.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.05406172799999, 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:47.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.048555599, 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.045717314, 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:47.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:47.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.04157347500001, 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.767 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.02733861900001, 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:47.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.016884163, 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.989174793, 2024-09-17T07:51:47.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:47.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.97463437099999, 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:47.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:47.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.970901236, 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.788 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.959859553, 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:47.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:47.792 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.945484767, 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.93152889300001, 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:47.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.927894809999998, 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:47.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:47.805 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.922888950000001, 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:47.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.855608787999998, 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:47.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:47.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.846684719999999, 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:47.814 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:47.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:47.819 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:47.819 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:47.821 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd1 2024-09-17T07:51:47.821 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:47.821 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:47.821 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:47.821 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:47.821 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd1'\'']/id' 2024-09-17T07:51:47.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:47.822 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:47.819+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:47.875 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=30941 2024-09-17T07:51:47.876 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 30941 2024-09-17T07:51:47.876 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 30941 -C rbd-nbd 2024-09-17T07:51:47.880 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:47.881 INFO:tasks.workunit.client.0.smithi098.stdout: 30941 ? 00:00:00 rbd-nbd 2024-09-17T07:51:47.881 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd feature enable rbd/testrbdnbd27356 journaling 2024-09-17T07:51:47.921 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:47.920+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:47.930 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:47.930 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:47.930 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:47.930 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:47.933 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-17T07:51:47.934 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:47.934 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:47.935 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:47.935 INFO:tasks.workunit.client.0.smithi098.stderr:++ which blkdiscard 2024-09-17T07:51:47.936 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-17T07:51:47.937 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:47.937 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-17T07:51:47.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:47.975 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd feature disable rbd/testrbdnbd27356 journaling 2024-09-17T07:51:48.022 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:48.021+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:48.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd1 30941 2024-09-17T07:51:48.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd1 2024-09-17T07:51:48.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=30941 2024-09-17T07:51:48.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:48.032 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:48.033 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:48.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:48.033 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:48.035 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:48.035 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:48.035 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:48.045 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:48.045 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:48.045 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:48.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:48.121 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:48.121 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^30941\b' 2024-09-17T07:51:48.121 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^30941\b' 2024-09-17T07:51:48.123 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:48.123+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.351734624999999, 2024-09-17T07:51:48.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:48.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.162 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.350241601999997, 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:48.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.345256018999997, 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:48.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.341116495999998, 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.171 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.336332175000003, 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:48.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:48.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.328560928000002, 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:48.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:48.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.318522686999998, 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:48.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:48.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.299205127, 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:48.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:48.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.286806818999999, 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:48.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.283155262999998, 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:48.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.282572111999997, 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:48.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.27798877, 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:48.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.277659718999999, 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:48.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:48.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.269460522999999, 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:48.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:48.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.268939275999998, 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.219 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.638652469, 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:48.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.224 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.614302668000001, 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:48.225 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:48.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:48.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.612775859000003, 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:48.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.233 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.581530852999997, 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:48.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:48.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 58.529707793999997, 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:48.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:48.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:48.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:48.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:48.243 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:48.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:48.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:48.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *30941\b' 2024-09-17T07:51:48.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *30941\b' 2024-09-17T07:51:48.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:48.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:48.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:48.246 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:48.243+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:48.344 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:48.344 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:48.344 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:48.344 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:48.345 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:48.344+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:48.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:48.446 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:48.445+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:48.449 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:48.449 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:48.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:48.547 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:48.545+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.079340999000003, 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.063721332999997, 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.059640510999998, 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00091687099999999998, 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536313+0000", 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536314+0000", 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3399999999999999e-07 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536322+0000", 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2300000000000008e-06 2024-09-17T07:51:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536332+0000", 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0098999999999999e-05 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536337+0000", 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4800000000000003e-06 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.536374+0000", 2024-09-17T07:51:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.7045999999999997e-05 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.537230+0000", 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00085638200000000004 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.053150369000001, 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010805350000000001, 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542803+0000", 2024-09-17T07:51:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542804+0000", 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.850000000000001e-07 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542816+0000", 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1459e-05 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542830+0000", 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3752e-05 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542837+0000", 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5950000000000001e-06 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.542886+0000", 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8646999999999997e-05 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.543884+0000", 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099809700000000009 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.047435159000003, 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.040197524, 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088098699999999996, 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555756+0000", 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555757+0000", 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7100000000000001e-07 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555767+0000", 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0519e-05 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555780+0000", 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555785+0000", 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4029999999999998e-06 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.555808+0000", 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3e-05 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.556637+0000", 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00082917600000000004 2024-09-17T07:51:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.032666919, 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011098950000000001, 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563287+0000", 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563296+0000", 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4090000000000003e-06 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563304+0000", 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3599999999999996e-06 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563317+0000", 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2853999999999999e-05 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.563349+0000", 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.1597999999999998e-05 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.564397+0000", 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001048034 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 59.028891684999998, 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010037819999999999, 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567062+0000", 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567063+0000", 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2900000000000003e-07 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567070+0000", 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6299999999999998e-06 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567080+0000", 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5289999999999997e-06 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567084+0000", 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0899999999999998e-06 2024-09-17T07:51:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.567131+0000", 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6894999999999997e-05 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.568066+0000", 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093500899999999999 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.077524879000002, 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.072816021000001, 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.068918969000002, 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.065329792999997, 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:48.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:48.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.646 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.060845475999997, 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:48.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:48.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.056913446000003, 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:48.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.655 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.053126607000003, 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.049474924999998, 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:48.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:48.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.045366483999999, 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:48.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:48.668 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.041766412999998, 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:48.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.038399820000002, 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:48.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:48.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:48.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:48.677 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.035008085999998, 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:48.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:48.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:48.681 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:48.683 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:48.682+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:48.683 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:51:48.683 INFO:tasks.workunit.client.0.smithi098.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-09-17T07:51:48.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:48.689 INFO:tasks.workunit.client.0.smithi098.stderr:+ EXPECTED=2 2024-09-17T07:51:48.690 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:51:48.690 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:48.690 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:48.691 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:48.691 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:48.693 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:48.693 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:48.693 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27356 2024-09-17T07:51:48.783 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:48.783+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd1 2024-09-17T07:51:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd1'\'']/id' 2024-09-17T07:51:48.798 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:48.798 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:48.799 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:48.799 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:48.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:48.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=31058 2024-09-17T07:51:48.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 31058 2024-09-17T07:51:48.865 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 31058 -C rbd-nbd 2024-09-17T07:51:48.870 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:48.870 INFO:tasks.workunit.client.0.smithi098.stdout: 31058 ? 00:00:00 rbd-nbd 2024-09-17T07:51:48.871 INFO:tasks.workunit.client.0.smithi098.stderr:++ ps -p 31058 -T 2024-09-17T07:51:48.871 INFO:tasks.workunit.client.0.smithi098.stderr:++ grep -c io_context_pool 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ ACTUAL=2 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 2 -eq 2 ']' 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd1 31058 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd1 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=31058 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:48.877 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:48.878 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:48.878 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:48.879 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:48.879 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:48.879 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:48.892 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:48.892+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:48.952 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:48.952 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:48.952 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:48.952 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:48.952 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:48.960 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:48.960 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^31058\b' 2024-09-17T07:51:48.960 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^31058\b' 2024-09-17T07:51:48.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:48.985 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:48.985+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:49.007 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:49.007 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:49.007 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *31058\b' 2024-09-17T07:51:49.007 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *31058\b' 2024-09-17T07:51:49.013 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:49.013 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:49.086 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:49.085+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:49.105 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:49.105 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:49.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.93823751599999, 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:49.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.930578, 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:49.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:49.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.92218843200001, 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:49.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:49.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:49.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:49.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.92181718200001, 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.123 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:49.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.91375421699999, 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:49.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.91338310500001, 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.132 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:49.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:49.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.90410270000001, 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.137 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:49.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.902443267, 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.896418442, 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.89616811499999, 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:49.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.88905427700001, 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:49.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.88853717800001, 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:49.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:49.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.168 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.86956601099999, 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:49.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:49.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.86330915100001, 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:49.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:49.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.856773089, 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:49.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.483906679, 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:49.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.48194627199999, 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:700 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 220.596277636, 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039903309999999997, 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 700 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507169+0000", 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.507170+0000", 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508024+0000", 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085453700000000005 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508036+0000", 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1516e-05 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508040+0000", 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3850000000000004e-06 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508070+0000", 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9963000000000001e-05 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508592+0000", 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052183799999999995 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.508659+0000", 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7589000000000006e-05 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511039+0000", 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023800190000000001 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511067+0000", 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.724e-05 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511122+0000", 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4854e-05 2024-09-17T07:51:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.511159+0000", 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7567999999999997e-05 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.28543199800001, 2024-09-17T07:51:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:97 3.0 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.28407055299999, 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042064320000000004, 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 97 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819376+0000", 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819377+0000", 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0720000000000001e-06 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819382+0000", 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5889999999999996e-06 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819396+0000", 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4279e-05 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819403+0000", 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0180000000000004e-06 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819919+0000", 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000516615 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820031+0000", 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111251 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.820172+0000", 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014081799999999999 2024-09-17T07:51:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823341+0000", 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031690770000000002 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823522+0000", 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000181502 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823563+0000", 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0611000000000001e-05 2024-09-17T07:51:49.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823582+0000", 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9599999999999999e-05 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.213 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:49.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:49.215 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:49.213+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:49.315 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:49.315+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:49.317 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:49.317 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:49.317 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:49.317 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:49.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:49.420 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:49.417+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:49.467 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:49.467 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:49.467 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:49.467 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:49.467 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:49.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:49.514 INFO:tasks.workunit.client.0.smithi098.stderr:+ EXPECTED=7 2024-09-17T07:51:49.514 INFO:tasks.workunit.client.0.smithi098.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd27356 2024-09-17T07:51:49.514 INFO:tasks.workunit.client.0.smithi098.stderr:++ local cmd 2024-09-17T07:51:49.514 INFO:tasks.workunit.client.0.smithi098.stderr:+++ id -u 2024-09-17T07:51:49.515 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' 1000 -eq 0 ']' 2024-09-17T07:51:49.515 INFO:tasks.workunit.client.0.smithi098.stderr:+++ which rbd 2024-09-17T07:51:49.517 INFO:tasks.workunit.client.0.smithi098.stderr:++ cmd=/usr/bin/rbd 2024-09-17T07:51:49.517 INFO:tasks.workunit.client.0.smithi098.stderr:++ shift 2024-09-17T07:51:49.517 INFO:tasks.workunit.client.0.smithi098.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd27356 2024-09-17T07:51:49.518 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:49.517+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:49.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 227.360257152, 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:49.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:49.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.99195153599999, 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:49.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.29976928100001, 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.628 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:49.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.293912425, 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.634 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:49.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.28221946599999, 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.640 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:49.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.213364883, 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:49.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:49.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:49.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:49.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.20146559400001, 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:49.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:49.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.15062104699999, 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:49.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.14461284000001, 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.664 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:49.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:49.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:49.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.669 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.13891248300001, 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:49.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:49.675 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.13317285799999, 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:49.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.80006180999999, 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:49.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:49.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:49.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.95496091300001, 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:49.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:49.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:49.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:49.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.94887419, 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:49.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:49.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.698 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.86541788900001, 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:49.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.859103457, 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:49.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.85273199700001, 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:49.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:49.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:49.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.84680952100001, 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.716 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:49.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 213.78882662500001, 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:49.722 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:49.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 209.83112195300001, 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:49.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:49.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:49.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:49.732 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:49.732 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:49.618+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:49.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:49.734 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:49.734 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:49.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T07:51:49.735 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:49.734+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:49.741 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:49.756 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:49.756 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:49.772 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:49.773 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:49.789 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:49.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:49.808 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:49.808 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:49.815 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV=/dev/nbd1 2024-09-17T07:51:49.815 INFO:tasks.workunit.client.0.smithi098.stderr:+ get_pid rbd 2024-09-17T07:51:49.815 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pool=rbd 2024-09-17T07:51:49.815 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns= 2024-09-17T07:51:49.815 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd device --device-type nbd --format xml list 2024-09-17T07:51:49.816 INFO:tasks.workunit.client.0.smithi098.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27356'\''][device='\''/dev/nbd1'\'']/id' 2024-09-17T07:51:49.838 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:49.838 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:49.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:49.863 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:49.863+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:49.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ PID=31127 2024-09-17T07:51:49.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ test -n 31127 2024-09-17T07:51:49.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -p 31127 -C rbd-nbd 2024-09-17T07:51:49.892 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-09-17T07:51:49.892 INFO:tasks.workunit.client.0.smithi098.stdout: 31127 ? 00:00:00 rbd-nbd 2024-09-17T07:51:49.893 INFO:tasks.workunit.client.0.smithi098.stderr:++ ps -p 31127 -T 2024-09-17T07:51:49.893 INFO:tasks.workunit.client.0.smithi098.stderr:++ grep -c io_context_pool 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ ACTUAL=7 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 7 -eq 7 ']' 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ unmap_device /dev/nbd1 31127 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ local args=/dev/nbd1 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ local pid=31127 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:49.899 INFO:tasks.workunit.client.0.smithi098.stderr:+ local cmd 2024-09-17T07:51:49.900 INFO:tasks.workunit.client.0.smithi098.stderr:++ id -u 2024-09-17T07:51:49.900 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 1000 -eq 0 ']' 2024-09-17T07:51:49.900 INFO:tasks.workunit.client.0.smithi098.stderr:++ which rbd 2024-09-17T07:51:49.901 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmd=/usr/bin/rbd 2024-09-17T07:51:49.902 INFO:tasks.workunit.client.0.smithi098.stderr:+ shift 2024-09-17T07:51:49.902 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-17T07:51:49.937 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:49.935+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:49.942 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:49.942 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:49.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:49.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd device --device-type nbd list 2024-09-17T07:51:49.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^31127\b' 2024-09-17T07:51:49.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^31127\b' 2024-09-17T07:51:50.038 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:50.036+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:50.038 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:50.038 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps -C rbd-nbd 2024-09-17T07:51:50.038 INFO:tasks.workunit.client.0.smithi098.stderr:+ expect_false grep '^ *31127\b' 2024-09-17T07:51:50.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^ *31127\b' 2024-09-17T07:51:50.044 INFO:tasks.workunit.client.0.smithi098.stderr:+ return 0 2024-09-17T07:51:50.044 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:50.089 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.089 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:50.139 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:50.138+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:50.236 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.236 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.239 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:50.238+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:50.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:50.339 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:50.338+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:50.383 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.383 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:50.475 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:50.474+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:50.528 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.529 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.537 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:50.545 INFO:tasks.workunit.client.0.smithi098.stderr:+ DEV= 2024-09-17T07:51:50.545 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-09-17T07:51:50.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-09-17T07:51:50.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ cleanup 2024-09-17T07:51:50.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ local ns s 2024-09-17T07:51:50.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ set +e 2024-09-17T07:51:50.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ mount 2024-09-17T07:51:50.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ fgrep /tmp/tmp.TyfTMF4aW5/mnt 2024-09-17T07:51:50.554 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm -Rf /tmp/tmp.TyfTMF4aW5 2024-09-17T07:51:50.562 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -n '' ']' 2024-09-17T07:51:50.563 INFO:tasks.workunit.client.0.smithi098.stderr:+ for ns in '' ${NS} 2024-09-17T07:51:50.563 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace '' status testrbdnbd27356 2024-09-17T07:51:50.581 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:50.581+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:50.602 INFO:tasks.workunit.client.0.smithi098.stdout:Watchers: none 2024-09-17T07:51:50.605 INFO:tasks.workunit.client.0.smithi098.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-17T07:51:50.605 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:50.641 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:50.641+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:50.649 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.649 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:50.679 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-09-17T07:51:50.742 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:50.741+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:50.782 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.782 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.790 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:50.844 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:50.843+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:50.891 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.892 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:50.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:50.945 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:50.944+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:50.998 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:50.999 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:51.048 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:51.047+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:51.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:51.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:51.089 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T07:51:51.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-17T07:51:51.106 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace '' status testrbdnbd27356 2024-09-17T07:51:51.106 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'Watchers: none' 2024-09-17T07:51:51.145 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.145 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.146 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:51.146+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:51.151 INFO:tasks.workunit.client.0.smithi098.stdout:Watchers: none 2024-09-17T07:51:51.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ break 2024-09-17T07:51:51.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd27356 2024-09-17T07:51:51.154 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:51.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-17T07:51:51.246 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:51.245+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:51.253 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.253 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:51.348 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:51.347+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:51.368 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.368 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.378 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:51.397 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-17T07:51:51.449 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:51.448+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:51.481 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.481 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:51.549 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:51.548+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:51.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-17T07:51:51.610 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.610 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:51.651 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:51.650+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:51.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-17T07:51:51.770 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.771 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.771 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:51.770+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:51.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:51.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:51.862 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:51.861+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:51.923 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:51.923 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:51.932 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:51.973 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:51.973+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:52.018 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:52.019 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:52.034 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:52.051 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:52.053 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:52.053+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:52.067 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 229.82007751200001, 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 224.451771896, 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.75958964099999, 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:630 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.75373278500001, 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034807330000000002, 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 630 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.323583+0000", 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3200000000000005e-07 2024-09-17T07:51:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324390+0000", 2024-09-17T07:51:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080675199999999999 2024-09-17T07:51:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324400+0000", 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5750000000000007e-06 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324404+0000", 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8970000000000001e-06 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324463+0000", 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9360000000000001e-05 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324523+0000", 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0387999999999998e-05 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.324597+0000", 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3798000000000001e-05 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.325942+0000", 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001344626 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.326994+0000", 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001052274 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327023+0000", 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8758000000000001e-05 2024-09-17T07:51:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.327063+0000", 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0673000000000002e-05 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.742039826, 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.67318524300001, 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:664 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.66128595399999, 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034795659999999999, 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 664 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416030+0000", 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416031+0000", 2024-09-17T07:51:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416866+0000", 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083558999999999999 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416878+0000", 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416883+0000", 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5029999999999999e-06 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416924+0000", 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1254999999999997e-05 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.416986+0000", 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1952999999999995e-05 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.417058+0000", 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2342000000000006e-05 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.418552+0000", 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014939879999999999 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419439+0000", 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088690700000000004 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419466+0000", 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6472e-05 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.419509+0000", 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3548999999999999e-05 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.610441407, 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.60443319999999, 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.59873284299999, 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.592993218, 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.143 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 219.25988217, 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:52.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:52.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.41478127299999, 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:52.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:52.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:52.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.40869455000001, 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.155 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.160 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.32523824899999, 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:52.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:52.166 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.31892381700001, 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:52.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.31255235699999, 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:52.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.30662988099999, 2024-09-17T07:51:52.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:52.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 216.24864698499999, 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:52.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:52.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 212.29094231299999, 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:52.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:52.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:52.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:52.194 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:52.195 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:52.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:52.196 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:52.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:52.197 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:52.195+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:52.296 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:52.296+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:52.306 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:52.306 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:52.306 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:52.306 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:52.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:52.397 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:52.397+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:52.412 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:52.412 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:52.413 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:52.413 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:52.413 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:52.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:52.498 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:52.497+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.35338776099999, 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.520 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:52.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.346048427, 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.525 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:52.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.34533135300001, 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.33610379199999, 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.32687922400001, 2024-09-17T07:51:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.311404584, 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.302115074, 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.29552482899999, 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.279342348, 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.27026742800001, 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.929828552, 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.92777627500001, 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.867544624, 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.70636547500001, 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.70634117399999, 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.70630679800001, 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.70628544100001, 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.703616797, 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.69389922400001, 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.72778883500001, 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:52.621 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:52.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:52.623 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:52.621+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:52.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:51:52.723 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:52.722+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:52.764 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:52.764 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:52.764 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:52.764 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:52.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:52.856 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:52.855+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:52.910 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:52.910 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:52.910 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:52.910 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:52.910 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:52.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:52.924 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:52.924+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:53.025 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:53.024+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:53.059 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:53.059 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.390396043, 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:53.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.38738317000001, 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:53.064 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:53.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.38511624500001, 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.382946627, 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:53.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:53.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.37965726, 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:53.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:53.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.374151131, 2024-09-17T07:51:53.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:53.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.371312846, 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:53.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.367169007, 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.090 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:53.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.352934151, 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.094 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:53.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.34247969499999, 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.314770325, 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.300229903, 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.109 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.296496768, 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:53.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:53.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.285455085, 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:53.113 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:53.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:53.117 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.271080299, 2024-09-17T07:51:53.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:53.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.257124425, 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:53.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.253490342000006, 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.127 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.130 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.248484481999995, 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.134 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.181204320000006, 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:53.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.172280251999993, 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.138 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:53.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.142 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:53.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:53.144 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:53.143+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:53.244 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:53.243+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:53.247 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:53.248 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:53.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:53.345 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:53.344+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:53.357 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:53.357 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:53.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:53.449 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:53.449+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:53.509 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.703135899000003, 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.701642875999994, 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.696657293000001, 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.692517769999995, 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.687733449000007, 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.679962201999999, 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.669923960999995, 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.650606401000005, 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.638208093000003, 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002276663, 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.869944+0000", 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0699999999999996e-07 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870913+0000", 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000968964 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870939+0000", 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519000000000001e-05 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.870950+0000", 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1420000000000001e-05 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871013+0000", 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2754000000000003e-05 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.871018+0000", 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1460000000000004e-06 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.872220+0000", 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001202053 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.634556536999995, 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.633973385999994, 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.629390044000004, 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.629060992999996, 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.620861797000003, 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 67.620340549999995, 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021911489999999999, 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887811+0000", 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887812+0000", 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.99e-07 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889121+0000", 2024-09-17T07:51:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001309302 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889136+0000", 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4351e-05 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889142+0000", 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3450000000000002e-06 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889192+0000", 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9906e-05 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.889196+0000", 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5700000000000003e-06 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890002+0000", 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080597600000000002 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.990053742999997, 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002242876, 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518098+0000", 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.518099+0000", 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519257+0000", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011578249999999999 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519275+0000", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8508e-05 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519283+0000", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519358+0000", 2024-09-17T07:51:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5321999999999995e-05 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.519367+0000", 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.132e-06 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.520341+0000", 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097366200000000003 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.965703941999998, 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.964177133, 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.932932127000001, 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 63.881109068000001, 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:53.593 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:53.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:53.596 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:53.594+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:53.693 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:53.693 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:53.693 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:53.695 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:51:53.695 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:53.695+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:53.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:53.797 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:53.795+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:53.802 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:53.802 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:53.802 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:53.802 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:53.802 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:53.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:53.897 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:53.896+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.433893437999998, 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.952 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:53.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:53.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.418273772000006, 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:53.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.401987598000005, 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:53.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:53.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.432077317999997, 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.964 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:53.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.427368459999997, 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:53.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:53.973 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.423471407999997, 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.419882231999999, 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:53.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:53.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.415397915, 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:53.982 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:53.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.986 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.411465884999998, 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:53.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.407679045999998, 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:53.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.404027364000001, 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:53.995 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:53.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:53.999 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.399918923000001, 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:54.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.396318852, 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:54.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.392952258999998, 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:54.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:54.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.389560525, 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:54.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.2.0:77 3.1 e163/127 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:7 v 163'106, mlcod=162'102)", 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.25530633699999999, 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029160549999999999, 2024-09-17T07:51:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.97e-07 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695204+0000", 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8959999999999996e-06 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695211+0000", 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695217+0000", 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.203e-06 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695259+0000", 2024-09-17T07:51:54.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2700000000000001e-05 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695284+0000", 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4643000000000001e-05 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698090+0000", 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028058620000000001 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698116+0000", 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5998000000000001e-05 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:468 3.2 e163/154 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:7 v 163'63, mlcod=162'60)", 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.25512059999999998, 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034243440000000002, 2024-09-17T07:51:54.022 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4999999999999999e-07 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695392+0000", 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8080000000000001e-06 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695401+0000", 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9809999999999992e-06 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695406+0000", 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.631e-06 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695441+0000", 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4848000000000002e-05 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695460+0000", 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8553e-05 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698778+0000", 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033180190000000002 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698810+0000", 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2254e-05 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.2.0:78 3.1 e163/127 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:7 v 163'108, mlcod=162'102)", 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.696589+0000", 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.25391677600000001, 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017333170000000001, 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:54.027 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696589+0000", 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696589+0000", 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696590+0000", 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6499999999999999e-07 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696596+0000", 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9800000000000003e-06 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696604+0000", 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7100000000000007e-06 2024-09-17T07:51:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696608+0000", 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1119999999999998e-06 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697020+0000", 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00041253200000000001 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697033+0000", 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2461e-05 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698299+0000", 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00126636 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698323+0000", 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3496999999999999e-05 2024-09-17T07:51:54.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:470 3.2 e163/154 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:7 v 163'65, mlcod=162'60)", 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.697259+0000", 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.25324723199999999, 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001703244, 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697259+0000", 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697259+0000", 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697260+0000", 2024-09-17T07:51:54.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697266+0000", 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000001e-06 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697280+0000", 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4283e-05 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697287+0000", 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4769999999999997e-06 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697323+0000", 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5281000000000003e-05 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697341+0000", 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7841000000000001e-05 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698939+0000", 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001598066 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698962+0000", 2024-09-17T07:51:54.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3553e-05 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.2.0:79 3.1 e163/127 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:7 v 163'110, mlcod=163'106)", 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.25109864999999998, 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001263007, 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:54.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:54.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.71e-07 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699423+0000", 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.446e-05 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699431+0000", 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4740000000000005e-06 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699436+0000", 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2710000000000003e-06 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699460+0000", 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4692999999999999e-05 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:54.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699474+0000", 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3528e-05 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.700645+0000", 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011708000000000001 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.700671+0000", 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5910000000000001e-05 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:54.042 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:54.042 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:54.043 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:54.042+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:54.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:54.144 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:54.143+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:54.149 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:54.149 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:54.149 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:54.150 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:54.158 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:54.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:51:54.245 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:54.245+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:54.304 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:54.304 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:54.304 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:54.304 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:54.304 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:54.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:54.317 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:54.318 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:54.334 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:54.334 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:54.346 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:54.346+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:54.350 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:54.350 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:54.365 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:54.366 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:54.381 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:54.398 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:54.416 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:54.416 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:54.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:54.466 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:54.466+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:54.549 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:54.548+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:54.567 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:54.567 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:54.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:54.650 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:54.650+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:54.677 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:54.677 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:54.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:54.700 INFO:tasks.workunit.client.0.smithi098.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-17T07:51:54.706 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:54.706+0000 7f78d684b640 0 -- 172.21.15.98:0/2050726329 >> [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] conn(0x7f78bc008980 msgr2=0x7f78bc00ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:51:54.712 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd27356 2024-09-17T07:51:54.769 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:54.769+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:54.807 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete...2024-09-17T07:51:54.806+0000 7f7efdfa2640 0 -- 172.21.15.98:0/3808747691 >> [v2:172.21.15.94:6808/31605,v1:172.21.15.94:6809/31605] conn(0x7f7ed8006840 msgr2=0x7f7ed8008c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-17T07:51:54.816 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 87% complete... Removing image: 93% complete...2024-09-17T07:51:54.816+0000 7f7efd7a1640 0 -- 172.21.15.98:0/3808747691 >> [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] conn(0x7f7ee003e910 msgr2=0x7f7ee0040d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:51:54.826 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-17T07:51:54.829 INFO:tasks.workunit.client.0.smithi098.stderr:+ for ns in '' ${NS} 2024-09-17T07:51:54.830 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace ns status testrbdnbd27356 2024-09-17T07:51:54.834 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:54.834 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:54.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:54.869 INFO:tasks.workunit.client.0.smithi098.stdout:Watchers: none 2024-09-17T07:51:54.872 INFO:tasks.workunit.client.0.smithi098.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-17T07:51:54.872 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 0.5 2024-09-17T07:51:54.884 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:54.884+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:54.943 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:54.944 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:54.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:54.952 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:54.951+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:55.047 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.047 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.053 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:55.052+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:55.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:55.154 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:55.153+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:55.194 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.194 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:55.255 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:55.254+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:55.340 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.341 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:55.355 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:55.355+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:55.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace ns status testrbdnbd27356 2024-09-17T07:51:55.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'Watchers: none' 2024-09-17T07:51:55.421 INFO:tasks.workunit.client.0.smithi098.stdout:Watchers: none 2024-09-17T07:51:55.421 INFO:tasks.workunit.client.0.smithi098.stderr:+ break 2024-09-17T07:51:55.421 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd27356 2024-09-17T07:51:55.447 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.447 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:55.457 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-17T07:51:55.456+0000 7ffa3aff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 69752) UID: 0 2024-09-17T07:51:55.552 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.553 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.558 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:55.557+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:55.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:55.659 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:55.657+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:55.674 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.674 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:55.707 INFO:tasks.workunit.client.0.smithi098.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-17T07:51:55.712 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:51:55.715 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd27356 2024-09-17T07:51:55.770 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:55.770+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:55.799 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.800 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:55.814 INFO:tasks.workunit.client.0.smithi098.stderr:2024-09-17T07:51:55.813+0000 7fc92304b640 0 -- 172.21.15.98:0/3310842225 >> [v2:172.21.15.26:6811/31886,v1:172.21.15.26:6813/31886] conn(0x7fc9040063d0 msgr2=0x7fc904008860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T07:51:55.829 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-17T07:51:55.829+0000 7fc92384c640 0 -- 172.21.15.98:0/3310842225 >> [v2:172.21.15.26:6808/31885,v1:172.21.15.26:6809/31885] conn(0x55ec332f1b10 msgr2=0x55ec332f3ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-17T07:51:55.839 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-17T07:51:55.842 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd namespace remove rbd/ns 2024-09-17T07:51:55.860 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:55.859+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:55.888 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd config global rm global rbd_default_pool 2024-09-17T07:51:55.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph osd pool delete new_default_pool27356 new_default_pool27356 --yes-i-really-really-mean-it 2024-09-17T07:51:55.955 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:55.955 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:55.961 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:55.959+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:55.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:56.061 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:56.061+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:56.062 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:56.062 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:56.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:56.162 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:56.162+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:56.167 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:56.167 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:56.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:56.263 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:56.262+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:56.273 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:56.273 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:56.282 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T07:51:56.365 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:56.364+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:56.386 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:56.386 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:56.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T07:51:56.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:51:56.466 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-09-17T07:51:56.465+0000 7f9906ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31600) UID: 0 2024-09-17T07:51:56.497 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-09-17T07:51:56.497 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:51:56.506 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T07:51:56.555 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-09-17T07:51:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:51:56.566 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:56.565+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:56.571 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-09-17T07:51:56.571 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:51:56.587 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-09-17T07:51:56.587 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:51:56.603 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-09-17T07:51:56.603 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:32 3.2 3:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.444574999, 2024-09-17T07:51:56.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020300992, 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165209+0000", 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.165210+0000", 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5000000000000002e-07 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172917+0000", 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077068919999999999 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172935+0000", 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8723999999999999e-05 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172945+0000", 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.825e-06 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.173015+0000", 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0036000000000005e-05 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185506+0000", 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012490696000000001 2024-09-17T07:51:56.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185510+0000", 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0690000000000003e-06 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:33 1.3 1:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.617 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.43691548300001, 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022353996000000001, 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172868+0000", 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.172870+0000", 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5030000000000001e-06 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181217+0000", 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083472919999999992 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181238+0000", 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.084e-05 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181245+0000", 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181303+0000", 2024-09-17T07:51:56.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8019999999999997e-05 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181433+0000", 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129367 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181552+0000", 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119737 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.185855+0000", 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043023979999999998 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195119+0000", 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092646020000000003 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195170+0000", 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0522999999999997e-05 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.195222+0000", 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2614000000000001e-05 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:34 1.3 1:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.42852591499999, 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023396041999999999, 2024-09-17T07:51:56.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181258+0000", 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181260+0000", 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188633+0000", 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073732399999999997 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188650+0000", 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7311000000000001e-05 2024-09-17T07:51:56.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188657+0000", 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6819999999999997e-06 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188693+0000", 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5487000000000001e-05 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188804+0000", 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011090999999999999 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.188909+0000", 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010555100000000001 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.193190+0000", 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042804139999999997 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204561+0000", 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01137178 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204620+0000", 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8383e-05 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.204654+0000", 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4368e-05 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:38 3.0 3:17e4f8bb:ns::rbd_data.12c1e94c40fb.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.42815466499999, 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031405362999999999, 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 38 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181629+0000", 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.181630+0000", 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189649+0000", 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080183129999999991 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189666+0000", 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7152000000000001e-05 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189674+0000", 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3750000000000003e-06 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189721+0000", 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6443000000000002e-05 2024-09-17T07:51:56.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213028+0000", 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023307101 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.213035+0000", 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8759999999999999e-06 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:40 3.0 3:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.4200917, 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046784315999999999, 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189692+0000", 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.189694+0000", 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4559999999999999e-06 2024-09-17T07:51:56.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197231+0000", 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075374429999999996 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197245+0000", 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3822000000000001e-05 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197251+0000", 2024-09-17T07:51:56.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5379999999999999e-06 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.197288+0000", 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6900999999999998e-05 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236473+0000", 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039184521 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236477+0000", 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6349999999999999e-06 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:39 1.5 1:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.41972058799999, 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013228491, 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.638 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190063+0000", 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.190064+0000", 2024-09-17T07:51:56.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6400000000000001e-07 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198758+0000", 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086937409999999996 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198770+0000", 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2383e-05 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198775+0000", 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198851+0000", 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5698000000000006e-05 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.198945+0000", 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4514999999999999e-05 2024-09-17T07:51:56.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203193+0000", 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042476190000000002 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203235+0000", 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2611000000000002e-05 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.203292+0000", 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6415000000000001e-05 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:42 1.5 1:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.41044018299999, 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012738454, 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.643 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199344+0000", 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.199345+0000", 2024-09-17T07:51:56.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2890000000000001e-06 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205135+0000", 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057904539999999996 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205154+0000", 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8658e-05 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205159+0000", 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.203e-06 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205191+0000", 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1442999999999997e-05 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.205297+0000", 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010648400000000001 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211882+0000", 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065848920000000002 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.211983+0000", 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101146 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.212082+0000", 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 3.0 3:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.40878075000001, 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021721154999999999, 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 44 2024-09-17T07:51:56.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201003+0000", 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.201004+0000", 2024-09-17T07:51:56.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.206984+0000", 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059798619999999999 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207004+0000", 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0058e-05 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207013+0000", 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207063+0000", 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0291999999999999e-05 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222721+0000", 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015658208 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222724+0000", 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8789999999999999e-06 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 3.0 3:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.40275592500001, 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029592054999999999, 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 47 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207028+0000", 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207029+0000", 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214506+0000", 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074765630000000003 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214525+0000", 2024-09-17T07:51:56.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8705000000000001e-05 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214533+0000", 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.089e-06 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214593+0000", 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9797999999999999e-05 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236618+0000", 2024-09-17T07:51:56.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022025698999999999 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236620+0000", 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.717e-06 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:44 1.4 1:3bb82aa7:ns::rbd_data.12c1e94c40fb.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.402505598, 2024-09-17T07:51:56.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015317562999999999, 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207278+0000", 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.207279+0000", 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208595+0000", 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013157209999999999 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208614+0000", 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9411000000000001e-05 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208622+0000", 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8399999999999995e-06 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208677+0000", 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4852000000000003e-05 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.208904+0000", 2024-09-17T07:51:56.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022703400000000001 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.209101+0000", 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019704400000000001 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214722+0000", 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056209880000000004 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222522+0000", 2024-09-17T07:51:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0077997930000000002 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222566+0000", 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3699000000000003e-05 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.222596+0000", 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0239999999999998e-05 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:48 1.3 1:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.39539176, 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029501709000000001, 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214392+0000", 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214394+0000", 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9350000000000001e-06 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223878+0000", 2024-09-17T07:51:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094840020000000001 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223890+0000", 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1843e-05 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223895+0000", 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999998e-06 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.223931+0000", 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6355e-05 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224035+0000", 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103471 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.224127+0000", 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1989000000000003e-05 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.228138+0000", 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040119170000000003 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243813+0000", 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015674595 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243864+0000", 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1081000000000001e-05 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.243894+0000", 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:47 1.4 1:350c28c3:ns::rbd_data.12c1e94c40fb.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.39487466099999, 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021629954999999999, 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214909+0000", 2024-09-17T07:51:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.214911+0000", 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.37e-06 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216107+0000", 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011966570000000001 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216131+0000", 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4134e-05 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216141+0000", 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9930000000000007e-06 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216231+0000", 2024-09-17T07:51:56.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9487999999999994e-05 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216412+0000", 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001816 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.216598+0000", 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018575500000000001 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.221512+0000", 2024-09-17T07:51:56.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049141669999999997 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236466+0000", 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014953126000000001 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236510+0000", 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4898999999999998e-05 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.236539+0000", 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8765999999999999e-05 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:54 3.2 3:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.375903494, 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016684925, 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 54 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233880+0000", 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.233883+0000", 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.278e-06 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238255+0000", 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043722930000000002 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238268+0000", 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3436e-05 2024-09-17T07:51:56.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238274+0000", 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.242e-06 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.238315+0000", 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1751000000000001e-05 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250561+0000", 2024-09-17T07:51:56.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012245424 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.250565+0000", 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5009999999999998e-06 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:58 3.2 3:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.36964663399999, 2024-09-17T07:51:56.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018182685000000001, 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4801", 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240137+0000", 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.240139+0000", 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6920000000000001e-06 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246614+0000", 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006474947 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246629+0000", 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500000000000001e-05 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246637+0000", 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246698+0000", 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0285999999999998e-05 2024-09-17T07:51:56.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258316+0000", 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011618267999999999 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.258320+0000", 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.963e-06 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4801.0:57 1.3 1:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.36311057200001, 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946779999999999, 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246673+0000", 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.246674+0000", 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0210000000000001e-06 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254740+0000", 2024-09-17T07:51:56.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080652610000000007 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254757+0000", 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7421000000000001e-05 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254766+0000", 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1710000000000008e-06 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.254832+0000", 2024-09-17T07:51:56.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5451999999999996e-05 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255000+0000", 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168188 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5", 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.255157+0000", 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015730400000000001 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.260305+0000", 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00514782 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264545+0000", 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042396339999999999 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264589+0000", 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3915999999999998e-05 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.264620+0000", 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1591999999999998e-05 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:457 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.99024416200001, 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016212842000000002, 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.687 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 457 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619540+0000", 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619541+0000", 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0249999999999999e-06 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619546+0000", 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.609e-06 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619569+0000", 2024-09-17T07:51:56.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2722e-05 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619575+0000", 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5079999999999996e-06 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619611+0000", 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6468999999999999e-05 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.619658+0000", 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6421999999999997e-05 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.631947+0000", 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012289817 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632020+0000", 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3169000000000004e-05 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635463+0000", 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034420589999999999 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635511+0000", 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7994000000000002e-05 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635753+0000", 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000242048 2024-09-17T07:51:56.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:458 3.0 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.988283755, 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014586032000000001, 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 458 2024-09-17T07:51:56.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621500+0000", 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621501+0000", 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.057e-06 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621506+0000", 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9450000000000001e-06 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621517+0000", 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0749e-05 2024-09-17T07:51:56.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621521+0000", 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1960000000000001e-06 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621558+0000", 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6844999999999997e-05 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.621600+0000", 2024-09-17T07:51:56.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2469000000000003e-05 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632043+0000", 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010442404000000001 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.632101+0000", 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8402000000000001e-05 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635777+0000", 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003675691 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635836+0000", 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9205e-05 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.636086+0000", 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025006900000000001 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:96 3.0 3:3cdec881:::rbd_data.123eb9c2b614.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.79176948099999, 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054563019999999997, 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 96 2024-09-17T07:51:56.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818014+0000", 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818015+0000", 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2300000000000001e-07 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818017+0000", 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5129999999999999e-06 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818024+0000", 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6980000000000001e-06 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818028+0000", 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2980000000000002e-06 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818444+0000", 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041509700000000001 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818503+0000", 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9608999999999998e-05 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.818588+0000", 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5185999999999994e-05 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821276+0000", 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026874640000000001 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823381+0000", 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002104992 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823435+0000", 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4592999999999997e-05 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.823471+0000", 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5329e-05 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.6934.0:41 3.0 3:0cbfebbd:::rbd_header.123eb9c2b614:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e162)", 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:52.689208+0000", 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.9205763259999999, 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004372707, 2024-09-17T07:51:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.6934", 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2050726329", 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 41 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.689208+0000", 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.689208+0000", 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.689210+0000", 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.091e-06 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.689217+0000", 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6399999999999997e-06 2024-09-17T07:51:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.689235+0000", 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7926000000000001e-05 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.689244+0000", 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.67e-06 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.690366+0000", 2024-09-17T07:51:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001122507 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.690511+0000", 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000144573 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.690984+0000", 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000473241 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.692296+0000", 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001311918 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.693501+0000", 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001204538 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.693535+0000", 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4177000000000002e-05 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:52.693580+0000", 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5426000000000002e-05 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.6934.0:56 3.0 3:0cbfebbd:::rbd_header.123eb9c2b614:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e163)", 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:53.695150+0000", 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.9146338979999999, 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0095258359999999993, 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.6934", 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2050726329", 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 56 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695150+0000", 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695150+0000", 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695152+0000", 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.593e-06 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695159+0000", 2024-09-17T07:51:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9569999999999996e-06 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695172+0000", 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.376e-05 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695181+0000", 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5029999999999997e-06 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.699604+0000", 2024-09-17T07:51:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044229300000000003 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.699735+0000", 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013142500000000001 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2", 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.700208+0000", 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000472676 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.700989+0000", 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000780772 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.704565+0000", 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035761320000000001 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.704625+0000", 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0628000000000003e-05 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.704676+0000", 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0460000000000001e-05 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:56.717 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:56.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-09-17T07:51:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:51:56.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T07:51:56.720 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-09-17T07:51:56.720 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:51:56.721 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:51:56.721 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:56.718+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:56.797 INFO:tasks.workunit.client.0.smithi098.stderr:pool 'new_default_pool27356' does not exist 2024-09-17T07:51:56.808 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T07:51:56.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T07:51:56.819 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:56.817+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:56.862 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:56.862 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:56.862 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:56.862 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 0, 3] 2024-09-17T07:51:56.862 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:51:56.862 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T07:51:56.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T07:51:56.870 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T07:51:56.873 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-09-17T07:51:56.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T07:51:56.955 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-17T07:51:56.955+0000 7fd81c5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31876) UID: 0 2024-09-17T07:51:57.019 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:57.020 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:57.021 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-09-17T07:51:57.020+0000 7f4ba13f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 61673) UID: 0 2024-09-17T07:51:57.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T07:51:57.121 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-09-17T07:51:57.121+0000 7f6b6d7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 57422) UID: 0 2024-09-17T07:51:57.145 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 166, 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "666af130-c9e0-433a-95ce-0b7c815e9e68", 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-17T07:46:41.695689+0000", 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-17T07:51:56.699721+0000", 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-09-17T07:50:48.503993+0000", 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-09-17T07:51:35.638903+0000", 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:51:57.146 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 7, 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:51:57.147 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 6, 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "quincy", 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.441250+0000", 2024-09-17T07:51:57.148 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8193, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:57.149 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:57.150 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "165", 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 13, 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 165, 2024-09-17T07:51:57.151 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [ 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": 3, 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": 3, 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:57.152 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:46:51.470493+0000", 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-09-17T07:51:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "14", 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cache", 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-09-17T07:47:00.696161+0000", 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 8201, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-09-17T07:51:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 4, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 4, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 4, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 4, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 4, 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-09-17T07:51:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "165", 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "20", 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "20", 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "20", 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-09-17T07:51:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 13, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 165, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": 1, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "writeback", 2024-09-17T07:51:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 250, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bloom", 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "target_size": 0, 2024-09-17T07:51:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "seed": 0 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 60, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 8, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [ 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: 1000000 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2024-09-17T07:51:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1", 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 38, 2024-09-17T07:51:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 121, 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 154, 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 39, 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 69763 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6801/69763", 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6803/69763", 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6807/69763", 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6805/69763", 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b26da222-1990-4e5d-a449-0ff9d561090f", 2024-09-17T07:51:57.175 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.8853302001953125, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 154, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:57.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.179 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:57.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31885 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6809/31885", 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6812/31885", 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6820/31885", 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6816/31885", 2024-09-17T07:51:57.182 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e6761959-1933-4d0a-a0ff-b437b983611e", 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:57.183 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 154, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.185 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-09-17T07:51:57.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31886 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6813/31886", 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6817/31886", 2024-09-17T07:51:57.189 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6823/31886", 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6821/31886", 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "5166dbfd-44ec-4d31-b486-b151dfde1443", 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:57.190 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 49, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 126, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 39, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 50, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.191 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6816", 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6817", 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6818", 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6819", 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.193 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6822", 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6823", 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6820", 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6821", 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 61678 2024-09-17T07:51:57.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6817/61678", 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6819/61678", 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6823/61678", 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6821/61678", 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f7829bfc-4efc-4bc9-8c4d-423690de31fd", 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 11, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 154, 2024-09-17T07:51:57.197 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6808", 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6809", 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6810", 2024-09-17T07:51:57.199 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6811", 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6814", 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6815", 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6812", 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6813", 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 31605 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6809/31605", 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6811/31605", 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6815/31605", 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6813/31605", 2024-09-17T07:51:57.203 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4e0679f8-a785-4804-93d3-79b7acabe009", 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-17T07:51:57.204 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 11, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 68, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 129, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 69, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6800", 2024-09-17T07:51:57.205 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6801", 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6802", 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6803", 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6806", 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6807", 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-09-17T07:51:57.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6804", 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.94:6805", 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 57427 2024-09-17T07:51:57.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.94:6801/57427", 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.94:6803/57427", 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.94:6807/57427", 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.94:6805/57427", 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:57.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.296842+0000", 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 39 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:57.213 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.328736+0000", 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:57.214 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.376227+0000", 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.427493+0000", 2024-09-17T07:51:57.215 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 50 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-09-17T07:51:57.216 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.355734+0000", 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138320759226367, 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-09-17T07:46:48.430328+0000", 2024-09-17T07:51:57.217 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 69 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.5", 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.1", 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": {}, 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-09-17T07:51:57.219 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [ 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-09-17T07:51:57.220 INFO:teuthology.orchestra.run.smithi026.stdout: "snaps": [ 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: "begin": 4, 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: "length": 5 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: "snaps": [ 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: "begin": 4, 2024-09-17T07:51:57.221 INFO:teuthology.orchestra.run.smithi026.stdout: "length": 5 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-09-17T07:51:57.222 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.223 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:57.226 INFO:tasks.thrashosds.thrasher:j is {'epoch': 166, 'fsid': '666af130-c9e0-433a-95ce-0b7c815e9e68', 'created': '2024-09-17T07:46:41.695689+0000', 'modified': '2024-09-17T07:51:56.699721+0000', 'last_up_change': '2024-09-17T07:50:48.503993+0000', 'last_in_change': '2024-09-17T07:51:35.638903+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-17T07:46:51.441250+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:46:51.470493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-17T07:47:00.696161+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '20', 'last_force_op_resend_prenautilus': '20', 'last_force_op_resend_preluminous': '20', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 38, 'up_from': 121, 'up_thru': 154, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6800', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6801', 'nonce': 69763}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 69763}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 69763}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 69763}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 69763}]}, 'public_addr': '172.21.15.26:6801/69763', 'cluster_addr': '172.21.15.26:6803/69763', 'heartbeat_back_addr': '172.21.15.26:6807/69763', 'heartbeat_front_addr': '172.21.15.26:6805/69763', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b26da222-1990-4e5d-a449-0ff9d561090f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8853302001953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 31885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6812', 'nonce': 31885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6820', 'nonce': 31885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 31885}, {'type': 'v1', 'addr': '172.21.15.26:6816', 'nonce': 31885}]}, 'public_addr': '172.21.15.26:6809/31885', 'cluster_addr': '172.21.15.26:6812/31885', 'heartbeat_back_addr': '172.21.15.26:6820/31885', 'heartbeat_front_addr': '172.21.15.26:6816/31885', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e6761959-1933-4d0a-a0ff-b437b983611e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6811', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 31886}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6815', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 31886}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 31886}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6819', 'nonce': 31886}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 31886}]}, 'public_addr': '172.21.15.26:6813/31886', 'cluster_addr': '172.21.15.26:6817/31886', 'heartbeat_back_addr': '172.21.15.26:6823/31886', 'heartbeat_front_addr': '172.21.15.26:6821/31886', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5166dbfd-44ec-4d31-b486-b151dfde1443', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 11, 'last_clean_end': 49, 'up_from': 126, 'up_thru': 39, 'down_at': 50, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6816', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6817', 'nonce': 61678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6818', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6819', 'nonce': 61678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6822', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6823', 'nonce': 61678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6820', 'nonce': 61678}, {'type': 'v1', 'addr': '172.21.15.94:6821', 'nonce': 61678}]}, 'public_addr': '172.21.15.94:6817/61678', 'cluster_addr': '172.21.15.94:6819/61678', 'heartbeat_back_addr': '172.21.15.94:6823/61678', 'heartbeat_front_addr': '172.21.15.94:6821/61678', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7829bfc-4efc-4bc9-8c4d-423690de31fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6808', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 31605}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6810', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 31605}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6814', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 31605}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6812', 'nonce': 31605}, {'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 31605}]}, 'public_addr': '172.21.15.94:6809/31605', 'cluster_addr': '172.21.15.94:6811/31605', 'heartbeat_back_addr': '172.21.15.94:6815/31605', 'heartbeat_front_addr': '172.21.15.94:6813/31605', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4e0679f8-a785-4804-93d3-79b7acabe009', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 68, 'up_from': 96, 'up_thru': 129, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6800', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 57427}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6802', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 57427}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6806', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 57427}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6804', 'nonce': 57427}, {'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 57427}]}, 'public_addr': '172.21.15.94:6801/57427', 'cluster_addr': '172.21.15.94:6803/57427', 'heartbeat_back_addr': '172.21.15.94:6807/57427', 'heartbeat_front_addr': '172.21.15.94:6805/57427', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.296842+0000', 'dead_epoch': 39}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.328736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.376227+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.427493+0000', 'dead_epoch': 50}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.355734+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T07:46:48.430328+0000', 'dead_epoch': 69}], 'pg_upmap': [{'pgid': '1.5', 'osds': [1, 0]}, {'pgid': '3.1', 'osds': [3, 2]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 1, 'snaps': [{'begin': 4, 'length': 5}]}, {'pool': 3, 'snaps': [{'begin': 4, 'length': 5}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:51:57.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:51:57.227 DEBUG:teuthology.parallel:result is None 2024-09-17T07:51:57.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.4801.0:60 1.7 e45/12 1:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head v 45'5, mlcod=45'3)", 2024-09-17T07:51:57.229 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 234.91437730800001, 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084874140000000004, 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257238+0000", 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.257240+0000", 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8419999999999999e-06 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261264+0000", 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040238030000000003 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261287+0000", 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3071e-05 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261296+0000", 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261342+0000", 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6019000000000001e-05 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.261361+0000", 2024-09-17T07:51:57.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8853999999999999e-05 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265680+0000", 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043191289999999997 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:02.265725+0000", 2024-09-17T07:51:57.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:56 3.0 e49/16 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:head v 49'27, mlcod=49'25)", 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 229.546071692, 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097168459999999995, 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.235 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625544+0000", 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.625545+0000", 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.544e-06 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627518+0000", 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019725820000000001 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627545+0000", 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7679e-05 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627555+0000", 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7429999999999997e-06 2024-09-17T07:51:57.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627612+0000", 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6518999999999999e-05 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.627633+0000", 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1296000000000001e-05 2024-09-17T07:51:57.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635225+0000", 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075918180000000002 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:07.635260+0000", 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5664999999999998e-05 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:628 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.85388943699999, 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034887780000000001, 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.241 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 628 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317726+0000", 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.317729+0000", 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6509999999999999e-06 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318682+0000", 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095249799999999997 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318694+0000", 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2323999999999999e-05 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318698+0000", 2024-09-17T07:51:57.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-06 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318725+0000", 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537e-05 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318783+0000", 2024-09-17T07:51:57.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7941e-05 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.318859+0000", 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6536e-05 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.320189+0000", 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001330007 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321145+0000", 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095558100000000005 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321171+0000", 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6101e-05 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.321215+0000", 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3432e-05 2024-09-17T07:51:57.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:634 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.836339622, 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034935650000000001, 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.248 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 634 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.335276+0000", 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8199999999999999e-07 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336081+0000", 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080487899999999999 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336093+0000", 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1895999999999999e-05 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336097+0000", 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2520000000000001e-06 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336123+0000", 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5975000000000001e-05 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336182+0000", 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8258999999999999e-05 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.336249+0000", 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6947999999999997e-05 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.337554+0000", 2024-09-17T07:51:57.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013050620000000001 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338621+0000", 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010674269999999999 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338700+0000", 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9293000000000001e-05 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.338769+0000", 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8792000000000001e-05 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:660 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.76748503900001, 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035910199999999999, 2024-09-17T07:51:57.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 660 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:57.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404130+0000", 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404131+0000", 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4900000000000005e-07 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404964+0000", 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083337299999999999 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404977+0000", 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.404982+0000", 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405027+0000", 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5504999999999999e-05 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405097+0000", 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9907999999999995e-05 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.405173+0000", 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6255999999999996e-05 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.406667+0000", 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493795 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407649+0000", 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098211900000000005 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407677+0000", 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7562000000000001e-05 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.407721+0000", 2024-09-17T07:51:57.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4304000000000001e-05 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:684 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.704741203, 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003505808, 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 684 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466874+0000", 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.466875+0000", 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467678+0000", 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000803482 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467689+0000", 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1077000000000001e-05 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467693+0000", 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7620000000000001e-06 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467729+0000", 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5565000000000003e-05 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467789+0000", 2024-09-17T07:51:57.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0951e-05 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.467868+0000", 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8552000000000003e-05 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.469292+0000", 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423821 2024-09-17T07:51:57.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470309+0000", 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001017476 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470336+0000", 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6415e-05 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.470380+0000", 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4109000000000001e-05 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:686 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.69873299599999, 2024-09-17T07:51:57.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036193950000000001, 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 686 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:57.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472882+0000", 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.472883+0000", 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.13e-07 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473710+0000", 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082752399999999995 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473720+0000", 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9049999999999992e-06 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473724+0000", 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4230000000000001e-06 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473788+0000", 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3973000000000006e-05 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473848+0000", 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0263000000000002e-05 2024-09-17T07:51:57.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.473915+0000", 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6712999999999996e-05 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.475368+0000", 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014532010000000001 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476431+0000", 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010629260000000001 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476459+0000", 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7722999999999999e-05 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.476502+0000", 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3130999999999998e-05 2024-09-17T07:51:57.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:688 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.69303263899999, 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035175620000000001, 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 688 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.478583+0000", 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479406+0000", 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082225799999999997 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479415+0000", 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479419+0000", 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4529999999999998e-06 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479478+0000", 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944999999999999e-05 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479532+0000", 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4181999999999997e-05 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.479597+0000", 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5618999999999996e-05 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.481090+0000", 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014921870000000001 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482028+0000", 2024-09-17T07:51:57.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093880099999999996 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482058+0000", 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9450000000000001e-05 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.482100+0000", 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2320999999999997e-05 2024-09-17T07:51:57.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4950.0:690 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 228.68729301400001, 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035163740000000001, 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4950", 2024-09-17T07:51:57.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 690 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484322+0000", 2024-09-17T07:51:57.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.484323+0000", 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485147+0000", 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082395199999999997 2024-09-17T07:51:57.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485156+0000", 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9879999999999993e-06 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485159+0000", 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3730000000000001e-06 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485218+0000", 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9002000000000002e-05 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485273+0000", 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4737000000000001e-05 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.485339+0000", 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5629000000000005e-05 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.486676+0000", 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013379119999999999 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487763+0000", 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010864309999999999 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487790+0000", 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6774000000000001e-05 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:08.487839+0000", 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8955999999999997e-05 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.5031.0:92 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 224.354181966, 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040542909999999998, 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.5031", 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817433+0000", 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817434+0000", 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817437+0000", 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0410000000000002e-06 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817445+0000", 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817450+0000", 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.335e-06 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817486+0000", 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6025999999999997e-05 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817545+0000", 2024-09-17T07:51:57.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8968000000000002e-05 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.817612+0000", 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7032999999999993e-05 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.819261+0000", 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016495539999999999 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821401+0000", 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021398329999999998 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821429+0000", 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7997999999999999e-05 2024-09-17T07:51:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:12.821488+0000", 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8437999999999999e-05 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:62 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.50908106899999, 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004090804, 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 62 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.298 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662534+0000", 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.662536+0000", 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3430000000000001e-06 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663492+0000", 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095610900000000002 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663506+0000", 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4827000000000001e-05 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663511+0000", 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9980000000000003e-06 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663559+0000", 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7259000000000003e-05 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663666+0000", 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107731 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.663806+0000", 2024-09-17T07:51:57.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013949700000000001 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.665755+0000", 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001949054 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666540+0000", 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078478 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666574+0000", 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4269e-05 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.666625+0000", 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0936999999999997e-05 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:64 3.1 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.50299434600001, 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036114599999999999, 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 64 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.305 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668621+0000", 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.668622+0000", 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9599999999999999e-07 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669463+0000", 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084133600000000004 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669475+0000", 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2421e-05 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669480+0000", 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2209999999999999e-06 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669529+0000", 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509000000000003e-05 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669649+0000", 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.669804+0000", 2024-09-17T07:51:57.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015504900000000001 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.671427+0000", 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00162305 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672157+0000", 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073058800000000001 2024-09-17T07:51:57.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672187+0000", 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9682000000000001e-05 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.672232+0000", 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5411999999999998e-05 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:99 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.419538045, 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037077820000000002, 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752077+0000", 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752078+0000", 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752912+0000", 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083329900000000002 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752923+0000", 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1739000000000001e-05 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752929+0000", 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4820000000000002e-06 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.752957+0000", 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8501999999999999e-05 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753020+0000", 2024-09-17T07:51:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670999999999997e-05 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.753102+0000", 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1773999999999993e-05 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.754400+0000", 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001298616 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755649+0000", 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012488149999999999 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755698+0000", 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8976e-05 2024-09-17T07:51:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.755785+0000", 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6910999999999995e-05 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:101 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.41322361300001, 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003503749, 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.758392+0000", 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9400000000000004e-07 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759188+0000", 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079579499999999997 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759199+0000", 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0993e-05 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759204+0000", 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3120000000000004e-06 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759240+0000", 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6826999999999998e-05 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759347+0000", 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106172 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.759473+0000", 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012634899999999999 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.760967+0000", 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001493959 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761819+0000", 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085260900000000005 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761854+0000", 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4085999999999999e-05 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.761895+0000", 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1853000000000001e-05 2024-09-17T07:51:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:103 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.40685215299999, 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036798249999999998, 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764763+0000", 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.764764+0000", 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1999999999999999e-07 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765622+0000", 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085813199999999995 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765631+0000", 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9530000000000005e-06 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765638+0000", 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765673+0000", 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5497000000000003e-05 2024-09-17T07:51:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765772+0000", 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8881000000000004e-05 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.765910+0000", 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013815600000000001 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.767742+0000", 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018317979999999999 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768373+0000", 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063104700000000003 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768402+0000", 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9048999999999999e-05 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.768443+0000", 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0819999999999999e-05 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:105 3.1 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.40092967699999, 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037373039999999999, 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105 2024-09-17T07:51:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.770686+0000", 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2200000000000004e-07 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771527+0000", 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000840845 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771538+0000", 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1430000000000001e-05 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771542+0000", 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5439999999999999e-06 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771575+0000", 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2625e-05 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771674+0000", 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9664e-05 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.771799+0000", 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125148 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.773343+0000", 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001543106 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774291+0000", 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094861900000000005 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774338+0000", 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6731999999999999e-05 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.774423+0000", 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4969000000000006e-05 2024-09-17T07:51:57.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.4886.0:130 3.1 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 221.34294678099999, 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035728779999999998, 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-09-17T07:51:57.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.4886", 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.98:0/2381877794", 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 130 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828668+0000", 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.828669+0000", 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4600000000000005e-07 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829511+0000", 2024-09-17T07:51:57.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084218499999999996 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829523+0000", 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1439e-05 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829526+0000", 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829574+0000", 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7911999999999998e-05 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829682+0000", 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107784 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4", 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.829815+0000", 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133168 2024-09-17T07:51:57.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.831377+0000", 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015618839999999999 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832107+0000", 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073004999999999999 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832155+0000", 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8046e-05 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:15.832241+0000", 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6173000000000003e-05 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(osd.1.11:72 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 217.38524210899999, 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036905200000000001, 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786373+0000", 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786374+0000", 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7700000000000005e-07 2024-09-17T07:51:57.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786377+0000", 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2669999999999998e-06 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786383+0000", 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3899999999999998e-06 2024-09-17T07:51:57.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786387+0000", 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.143e-06 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786882+0000", 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049448500000000002 2024-09-17T07:51:57.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.786926+0000", 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4518000000000003e-05 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:48:19.790064+0000", 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031372399999999999 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:469 3.0 e163/108 3:0b0c2ebb:::rbd_data.123eb9c2b614.0000000000000006:7 v 163'403, mlcod=162'400)", 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:53.695484+0000", 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.476131606, 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0058932300000000002, 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695484+0000", 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695484+0000", 2024-09-17T07:51:57.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695484+0000", 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9999999999999999e-07 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695489+0000", 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9919999999999998e-06 2024-09-17T07:51:57.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695499+0000", 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8349999999999999e-06 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695505+0000", 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1589999999999998e-06 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695570+0000", 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5519999999999996e-05 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.695590+0000", 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9595000000000001e-05 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.701350+0000", 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057600619999999998 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.701377+0000", 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6767e-05 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.1.0:471 3.0 e163/108 3:2cbaf2f0:::rbd_data.123eb9c2b614.000000000000000c:7 v 163'405, mlcod=162'400)", 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:53.697583+0000", 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.4740322479999999, 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00381628, 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.697583+0000", 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.697583+0000", 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:57.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.697584+0000", 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9500000000000001e-07 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.697593+0000", 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6379999999999993e-06 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.697610+0000", 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7682000000000001e-05 2024-09-17T07:51:57.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.697619+0000", 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4780000000000008e-06 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.698032+0000", 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041341199999999999 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.698058+0000", 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6264000000000001e-05 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.701385+0000", 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033266519999999998 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:57.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.701399+0000", 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4259e-05 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:57.363 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:57.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T07:51:57.364 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:57.224+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:57.366 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-17T07:51:57.366+0000 7f5006ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-09-17T07:51:57.389 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T07:51:57.390 DEBUG:teuthology.orchestra.run.smithi098:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T07:51:57.448 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T07:51:57.448 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-17T07:51:57.458 INFO:tasks.thrashosds:joining thrashosds 2024-09-17T07:51:57.465 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:57.466 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:57.466 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:57.466 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:57.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T07:51:57.571 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:57.571 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:57.572 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:57.572 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:57.572 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:57.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T07:51:57.635 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:51:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:51:57.648 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-17T07:51:57.470262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101254799,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1121,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":364972119,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1279,"ondisk_log_size":1279,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1830504,"kb_used_data":304260,"kb_used_omap":15,"kb_used_meta":1513008,"kb_avail":470028696,"statfs":{"total":483183820800,"available":481309384704,"internally_reserved":0,"allocated":311562240,"data_stored":309720843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15949,"internal_metadata":1549320627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-128493860,"num_objects":-31,"num_object_clones":-19,"num_object_copies":-62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":26,"num_read":438,"num_read_kb":380,"num_write":101,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13,"num_bytes_recovered":-137,"num_keys_recovered":-26,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000779"},"pg_stats":[{"pgid":"1.0","version":"156'10","reported_seq":84,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:51:51.687682+0000","last_change":"2024-09-17T07:51:51.687881+0000","last_active":"2024-09-17T07:51:51.687682+0000","last_peered":"2024-09-17T07:51:51.687682+0000","last_clean":"2024-09-17T07:51:51.687682+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:51:51.687682+0000","last_undegraded":"2024-09-17T07:51:51.687682+0000","last_fullsized":"2024-09-17T07:51:51.687682+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:52.524331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064497399999999998,"stat_sum":{"num_bytes":16777370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":9,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.2","version":"165'73","reported_seq":845,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.884448+0000","last_change":"2024-09-17T07:51:55.705959+0000","last_active":"2024-09-17T07:51:55.884448+0000","last_peered":"2024-09-17T07:51:55.884448+0000","last_clean":"2024-09-17T07:51:55.884448+0000","last_became_active":"2024-09-17T07:51:36.653275+0000","last_became_peered":"2024-09-17T07:51:36.653275+0000","last_unstale":"2024-09-17T07:51:55.884448+0000","last_undegraded":"2024-09-17T07:51:55.884448+0000","last_fullsized":"2024-09-17T07:51:55.884448+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.123023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":37815682,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.1","version":"54'4","reported_seq":120,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:51:51.687300+0000","last_change":"2024-09-17T07:51:51.687677+0000","last_active":"2024-09-17T07:51:51.687300+0000","last_peered":"2024-09-17T07:51:51.687300+0000","last_clean":"2024-09-17T07:51:51.687300+0000","last_became_active":"2024-09-17T07:51:36.648087+0000","last_became_peered":"2024-09-17T07:51:36.648087+0000","last_unstale":"2024-09-17T07:51:51.687300+0000","last_undegraded":"2024-09-17T07:51:51.687300+0000","last_fullsized":"2024-09-17T07:51:51.687300+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:03.506247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001131167,"stat_sum":{"num_bytes":8388684,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.3","version":"165'487","reported_seq":2195,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.838038+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:51:55.838038+0000","last_peered":"2024-09-17T07:51:55.838038+0000","last_clean":"2024-09-17T07:51:55.838038+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:51:55.838038+0000","last_undegraded":"2024-09-17T07:51:55.838038+0000","last_fullsized":"2024-09-17T07:51:55.838038+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.2","version":"165'7","reported_seq":74,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.701133+0000","last_change":"2024-09-17T07:51:55.701133+0000","last_active":"2024-09-17T07:51:55.701133+0000","last_peered":"2024-09-17T07:51:55.701133+0000","last_clean":"2024-09-17T07:51:55.701133+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:51:55.701133+0000","last_undegraded":"2024-09-17T07:51:55.701133+0000","last_fullsized":"2024-09-17T07:51:55.701133+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:55:18.817814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"3.0","version":"165'460","reported_seq":2289,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.885914+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:51:55.885914+0000","last_peered":"2024-09-17T07:51:55.885914+0000","last_clean":"2024-09-17T07:51:55.885914+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:51:55.885914+0000","last_undegraded":"2024-09-17T07:51:55.885914+0000","last_fullsized":"2024-09-17T07:51:55.885914+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.3","version":"165'16","reported_seq":309,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.825170+0000","last_change":"2024-09-17T07:51:55.704885+0000","last_active":"2024-09-17T07:51:55.825170+0000","last_peered":"2024-09-17T07:51:55.825170+0000","last_clean":"2024-09-17T07:51:55.825170+0000","last_became_active":"2024-09-17T07:51:36.652299+0000","last_became_peered":"2024-09-17T07:51:36.652299+0000","last_unstale":"2024-09-17T07:51:55.825170+0000","last_undegraded":"2024-09-17T07:51:55.825170+0000","last_fullsized":"2024-09-17T07:51:55.825170+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:34:56.626824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":75497548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.0","version":"13'72","reported_seq":225,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.704594+0000","last_change":"2024-09-17T07:50:49.510685+0000","last_active":"2024-09-17T07:51:55.704594+0000","last_peered":"2024-09-17T07:51:55.704594+0000","last_clean":"2024-09-17T07:51:55.704594+0000","last_became_active":"2024-09-17T07:50:49.510352+0000","last_became_peered":"2024-09-17T07:50:49.510352+0000","last_unstale":"2024-09-17T07:51:55.704594+0000","last_undegraded":"2024-09-17T07:51:55.704594+0000","last_fullsized":"2024-09-17T07:51:55.704594+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.892029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":400,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.836879+0000","last_change":"2024-09-17T07:51:55.701579+0000","last_active":"2024-09-17T07:51:55.836879+0000","last_peered":"2024-09-17T07:51:55.836879+0000","last_clean":"2024-09-17T07:51:55.836879+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:51:55.836879+0000","last_undegraded":"2024-09-17T07:51:55.836879+0000","last_fullsized":"2024-09-17T07:51:55.836879+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:36.393955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.4","version":"165'10","reported_seq":289,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.822803+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:51:55.822803+0000","last_peered":"2024-09-17T07:51:55.822803+0000","last_clean":"2024-09-17T07:51:55.822803+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:51:55.822803+0000","last_undegraded":"2024-09-17T07:51:55.822803+0000","last_fullsized":"2024-09-17T07:51:55.822803+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.5","version":"165'9","reported_seq":200,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.881550+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:51:55.881550+0000","last_peered":"2024-09-17T07:51:55.881550+0000","last_clean":"2024-09-17T07:51:55.881550+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:51:55.881550+0000","last_undegraded":"2024-09-17T07:51:55.881550+0000","last_fullsized":"2024-09-17T07:51:55.881550+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.6","version":"45'3","reported_seq":75,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:51:51.687349+0000","last_change":"2024-09-17T07:51:51.687604+0000","last_active":"2024-09-17T07:51:51.687349+0000","last_peered":"2024-09-17T07:51:51.687349+0000","last_clean":"2024-09-17T07:51:51.687349+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:51:51.687349+0000","last_undegraded":"2024-09-17T07:51:51.687349+0000","last_fullsized":"2024-09-17T07:51:51.687349+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:35.978441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065280999999999998,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"1.7","version":"45'5","reported_seq":196,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-17T07:51:55.701389+0000","last_change":"2024-09-17T07:51:55.701389+0000","last_active":"2024-09-17T07:51:55.701389+0000","last_peered":"2024-09-17T07:51:55.701389+0000","last_clean":"2024-09-17T07:51:55.701389+0000","last_became_active":"2024-09-17T07:51:36.647444+0000","last_became_peered":"2024-09-17T07:51:36.647444+0000","last_unstale":"2024-09-17T07:51:55.701389+0000","last_undegraded":"2024-09-17T07:51:55.701389+0000","last_fullsized":"2024-09-17T07:51:55.701389+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:09:00.276726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25165932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":104924973,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":82366464,"data_stored":86188390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":100663909,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":66,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":260047146,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285339648,"data_stored":285214066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":0,"seq":412316860441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640320,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328976,"kb_used_data":10156,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94042864,"statfs":{"total":96636764160,"available":96299892736,"internally_reserved":0,"allocated":10399744,"data_stored":9261621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.318},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.42899999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.378}]},{"osd":2,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.36599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.34000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.29599999999999999}]}]},{"osd":3,"up_from":126,"seq":541165879310,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415748,"kb_used_data":117812,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":93956092,"statfs":{"total":96636764160,"available":96211038208,"internally_reserved":0,"allocated":120639488,"data_stored":119478164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.443},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.375},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.40200000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.27600000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.41899999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319948,"kb_used_data":14260,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051892,"statfs":{"total":96636764160,"available":96309137408,"internally_reserved":0,"allocated":14602240,"data_stored":13456124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.24199999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28899999999999998,"15min":0.31900000000000001},"min":{"1min":0.19700000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.35299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.40000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.27700000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.435}]},{"osd":4,"last update":"Tue Sep 17 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.628,"15min":0.628},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42599999999999999,"15min":0.46000000000000002},"min":{"1min":0.30199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.52000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.497}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353296,"kb_used_data":47620,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94018544,"statfs":{"total":96636764160,"available":96274989056,"internally_reserved":0,"allocated":48762880,"data_stored":47601125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.246},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.29399999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.309,"15min":0.33500000000000002},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.38100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.35399999999999998},"min":{"1min":0.254,"5min":0.222,"15min":0.222},"max":{"1min":0.48499999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.314}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.44},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.58299999999999996}]},{"osd":4,"last update":"Tue Sep 17 07:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53600000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.51100000000000001},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.625,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.495}]},{"osd":5,"last update":"Tue Sep 17 07:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.45100000000000001}]}]},{"osd":0,"up_from":121,"seq":519691042831,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412536,"kb_used_data":114412,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":93959304,"statfs":{"total":96636764160,"available":96214327296,"internally_reserved":0,"allocated":117157888,"data_stored":119923809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.309},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.65500000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.58799999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67137536,"data_stored":67109151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46153728,"data_stored":46137616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92311552,"data_stored":92275168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43880448,"data_stored":47772893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25788416,"data_stored":25765148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12664832,"data_stored":12649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:51:57.649 INFO:tasks.thrashosds.thrasher:Setting 1.2 to [3, 1] 2024-09-17T07:51:57.650 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.2', '3', '1'] 2024-09-17T07:51:57.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.2 3 1 2024-09-17T07:51:57.679 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:33 3.3 3:cdd39f54:ns::rbd_data.12c1e94c40fb.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.51328586899999, 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.030622203000000001, 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165226+0000", 2024-09-17T07:51:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.165227+0000", 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3300000000000001e-07 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172505+0000", 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072782879999999999 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172519+0000", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3952e-05 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172523+0000", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6360000000000002e-06 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172606+0000", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2236000000000003e-05 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195844+0000", 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.023238828 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.195848+0000", 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5300000000000001e-06 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:34 3.3 3:d2a9fadb:ns::rbd_data.12c1e94c40fb.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.50594653499999, 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.032201267999999998, 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172565+0000", 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.172566+0000", 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2589999999999999e-06 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179356+0000", 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067896839999999998 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179383+0000", 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7005999999999999e-05 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179394+0000", 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1394e-05 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179449+0000", 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4673999999999999e-05 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204764+0000", 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025314861000000001 2024-09-17T07:51:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.204766+0000", 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.39e-06 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:32 1.2 1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.505229461, 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012117618, 2024-09-17T07:51:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173282+0000", 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.173283+0000", 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.268e-06 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179914+0000", 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0066302189999999997 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179929+0000", 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5753999999999999e-05 2024-09-17T07:51:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179937+0000", 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.498e-06 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.179980+0000", 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2765000000000001e-05 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.180097+0000", 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000117314 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185272+0000", 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0051745849999999998 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185326+0000", 2024-09-17T07:51:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4815999999999998e-05 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.185400+0000", 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3399000000000007e-05 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:39 3.1 3:b69be043:ns::rbd_data.12c1e94c40fb.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.49600190000001, 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020907184999999998, 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 39 2024-09-17T07:51:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182510+0000", 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.182511+0000", 2024-09-17T07:51:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0410000000000001e-06 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189594+0000", 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070828449999999999 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189621+0000", 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.722e-05 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189651+0000", 2024-09-17T07:51:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9720999999999999e-05 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.189701+0000", 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0167999999999998e-05 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203412+0000", 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013710953999999999 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.203417+0000", 2024-09-17T07:51:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2360000000000004e-06 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:42 3.1 3:b86c84fb:ns::rbd_data.12c1e94c40fb.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.486777332, 2024-09-17T07:51:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020911011, 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 42 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191734+0000", 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.191736+0000", 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.751e-06 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198943+0000", 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072064779999999997 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198962+0000", 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9774000000000001e-05 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.198970+0000", 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.9359999999999999e-06 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.199025+0000", 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5081999999999999e-05 2024-09-17T07:51:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212642+0000", 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013616964 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.212645+0000", 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.0259999999999999e-06 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:48 3.3 3:c1a66aa3:ns::rbd_data.12c1e94c40fb.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.47130269199999, 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037832497, 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 48 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207209+0000", 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.207210+0000", 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3349999999999999e-06 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213954+0000", 2024-09-17T07:51:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067431890000000001 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213970+0000", 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6243999999999999e-05 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.213978+0000", 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3799999999999994e-06 2024-09-17T07:51:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.214033+0000", 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4763999999999999e-05 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245039+0000", 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031005721 2024-09-17T07:51:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.245041+0000", 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.864e-06 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:51 3.1 3:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.46201318199999, 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.024704092, 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 51 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216498+0000", 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.216501+0000", 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5459999999999998e-06 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223050+0000", 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065494139999999999 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223069+0000", 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8964999999999999e-05 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223077+0000", 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5850000000000002e-06 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223130+0000", 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.2964999999999999e-05 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241200+0000", 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018070016000000001 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.241203+0000", 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6010000000000002e-06 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:52 3.1 3:90adac31:ns::rbd_data.12c1e94c40fb.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.45542293700001, 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020965557999999999, 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 52 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223089+0000", 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.223090+0000", 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5149999999999999e-06 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229411+0000", 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0063210109999999996 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229450+0000", 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8418999999999999e-05 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229463+0000", 2024-09-17T07:51:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3491e-05 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.229549+0000", 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5439000000000006e-05 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244052+0000", 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014503399 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.244054+0000", 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2840000000000001e-06 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:57 3.3 3:df196404:ns::rbd_data.12c1e94c40fb.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.43924045599999, 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025498885999999998, 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 57 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239271+0000", 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.239272+0000", 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0300000000000001e-06 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246325+0000", 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0070528220000000003 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246339+0000", 2024-09-17T07:51:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4307999999999999e-05 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246347+0000", 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3100000000000003e-06 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.246392+0000", 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5702999999999999e-05 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264767+0000", 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.018374669 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.264770+0000", 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.044e-06 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4801.0:60 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T07:51:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.430165536, 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017650266000000001, 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4801", 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/2630538197", 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 60 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248346+0000", 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.248347+0000", 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.037e-06 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254611+0000", 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062643200000000003 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254634+0000", 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2215999999999999e-05 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254642+0000", 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4789999999999996e-06 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.254689+0000", 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6779000000000001e-05 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265994+0000", 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011305079000000001 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:02.265996+0000", 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.356e-06 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:424 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.08972666, 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013984539000000001, 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 424 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588785+0000", 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588786+0000", 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8700000000000005e-07 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588789+0000", 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5209999999999998e-06 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588800+0000", 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1062e-05 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588805+0000", 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588852+0000", 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6384999999999997e-05 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.588899+0000", 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7034999999999999e-05 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600890+0000", 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011991725 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.600977+0000", 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6138999999999996e-05 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602458+0000", 2024-09-17T07:51:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014815469999999999 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602512+0000", 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4188999999999999e-05 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602770+0000", 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025719000000000002 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:425 3.3 3:e1a15f81:::rbd_data.123eb9c2b614.000000000000000b:head [sparse-read 135168~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.08767438300001, 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012263631000000001, 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 425 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590837+0000", 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590838+0000", 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.048e-06 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590843+0000", 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9420000000000003e-06 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590855+0000", 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1798e-05 2024-09-17T07:51:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590864+0000", 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5690000000000005e-06 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590898+0000", 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4091000000000003e-05 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.590925+0000", 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7146999999999999e-05 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601006+0000", 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010081087000000001 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.601094+0000", 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7985000000000005e-05 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602789+0000", 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016946299999999999 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.602838+0000", 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9620999999999998e-05 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.603101+0000", 2024-09-17T07:51:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026271299999999998 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4950.0:490 3.3 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.027442732, 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013632093, 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4950", 2024-09-17T07:51:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/846670005", 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 490 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651069+0000", 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651070+0000", 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5600000000000004e-07 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651075+0000", 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0429999999999998e-06 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651084+0000", 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5599999999999999e-06 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651090+0000", 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0930000000000002e-06 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651125+0000", 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5867000000000001e-05 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.651159+0000", 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3300999999999998e-05 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662655+0000", 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011496454999999999 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.662742+0000", 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6570999999999995e-05 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664402+0000", 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001659982 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664453+0000", 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0769000000000001e-05 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:07.664701+0000", 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024849600000000001 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.744 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:85 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.86626358300001, 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012264974, 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.745 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812248+0000", 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.84e-07 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812250+0000", 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.906e-06 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812259+0000", 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812268+0000", 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7940000000000008e-06 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820222+0000", 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0079543370000000006 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.820330+0000", 2024-09-17T07:51:57.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010777600000000001 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824421+0000", 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0040907799999999996 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824457+0000", 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6727999999999998e-05 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824513+0000", 2024-09-17T07:51:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5733999999999999e-05 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:86 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.86623928200001, 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012345253, 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812272+0000", 2024-09-17T07:51:57.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812273+0000", 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.7399999999999999e-07 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812275+0000", 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.03e-06 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812293+0000", 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.874e-05 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812296+0000", 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.9440000000000001e-06 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821366+0000", 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0090697209999999993 2024-09-17T07:51:57.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821426+0000", 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9551e-05 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824521+0000", 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030954089999999999 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824545+0000", 2024-09-17T07:51:57.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3765999999999999e-05 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824618+0000", 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2818000000000007e-05 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:87 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:57.755 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.86620490600001, 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013992821000000001, 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812307+0000", 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.36e-07 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812312+0000", 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4240000000000001e-06 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812317+0000", 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8799999999999999e-06 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812324+0000", 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8109999999999997e-06 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821888+0000", 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095635159999999993 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.821948+0000", 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0365000000000002e-05 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826230+0000", 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0042823510000000002 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826257+0000", 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6857000000000001e-05 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826300+0000", 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2381e-05 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:88 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.866183549, 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014038082, 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 88 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812328+0000", 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4200000000000002e-07 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812330+0000", 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.579e-06 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812334+0000", 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5299999999999998e-06 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.812338+0000", 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.078e-06 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822550+0000", 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.01021274 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.822633+0000", 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2465000000000004e-05 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826306+0000", 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036727639999999998 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826328+0000", 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2425e-05 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.826366+0000", 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8259000000000001e-05 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:89 3.3 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.86351490499999, 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013484612, 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 89 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814997+0000", 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.814998+0000", 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815001+0000", 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.614e-06 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815010+0000", 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.2380000000000003e-06 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.815015+0000", 2024-09-17T07:51:57.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1609999999999996e-06 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823115+0000", 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0081006350000000001 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.823180+0000", 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4781000000000002e-05 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828421+0000", 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0052408790000000004 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828446+0000", 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5188999999999999e-05 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.828481+0000", 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5287999999999998e-05 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5031.0:110 3.3 3:e63daa6a:::rbd_object_map.123eb9c2b614:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.853797332, 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.012135249000000001, 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5031", 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/3173719775", 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 110 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.771 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824714+0000", 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824715+0000", 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0900000000000001e-07 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824725+0000", 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.679e-06 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824733+0000", 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.5580000000000001e-06 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.824744+0000", 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0909e-05 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833532+0000", 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0087877830000000004 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.833589+0000", 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6628000000000001e-05 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836775+0000", 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.003186758 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836814+0000", 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8361000000000001e-05 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:12.836850+0000", 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5964000000000003e-05 2024-09-17T07:51:57.775 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.5197.0:19 3.3 3:eb262044:ns::rbd_data.12c1e94c40fb.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.88768694300001, 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.013322706, 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.5197", 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "172.21.15.98:0/332757234", 2024-09-17T07:51:57.776 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 19 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790825+0000", 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790826+0000", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.026e-06 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790831+0000", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9150000000000004e-06 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790844+0000", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2860999999999999e-05 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:57.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790849+0000", 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.215e-06 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790881+0000", 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2385000000000002e-05 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for blocked object", 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.790910+0000", 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8836999999999999e-05 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800184+0000", 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0092736910000000006 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for rw locks", 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.800296+0000", 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000112194 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.803912+0000", 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0036159899999999999 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804007+0000", 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.5409999999999996e-05 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:48:19.804147+0000", 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00014018200000000001 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:57.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:57.782 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:57.783 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T07:51:57.879 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:57.880 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:57.880 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:57.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T07:51:57.984 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:57.985 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:57.985 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:57.985 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:57.985 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:57.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.421651604, 2024-09-17T07:51:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020579190000000001, 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668357+0000", 2024-09-17T07:51:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.668358+0000", 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.2699999999999998e-07 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669254+0000", 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00089603000000000003 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669267+0000", 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3704e-05 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669275+0000", 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3259999999999998e-06 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669320+0000", 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5408999999999998e-05 2024-09-17T07:51:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.669326+0000", 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7969999999999997e-06 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.670415+0000", 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001088826 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.418638731, 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001296061, 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671370+0000", 2024-09-17T07:51:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6199999999999997e-07 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671931+0000", 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00056068100000000005 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671944+0000", 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2731000000000001e-05 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671950+0000", 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1659999999999999e-06 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.671999+0000", 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8828000000000001e-05 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672003+0000", 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1450000000000001e-06 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.672666+0000", 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000662748 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.416371806, 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001234625, 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673636+0000", 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.673637+0000", 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9800000000000002e-07 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674176+0000", 2024-09-17T07:51:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053858400000000005 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674189+0000", 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3339999999999999e-05 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674196+0000", 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.3709999999999996e-06 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674244+0000", 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7899000000000003e-05 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674248+0000", 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3529999999999997e-06 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.674871+0000", 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00062317999999999998 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.414202188, 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013693010000000001, 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675806+0000", 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.675807+0000", 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4600000000000003e-07 2024-09-17T07:51:58.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676344+0000", 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00053732400000000005 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676358+0000", 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3682999999999999e-05 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676364+0000", 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2199999999999997e-06 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676411+0000", 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.676415+0000", 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0620000000000002e-06 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.677175+0000", 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00076004800000000004 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.410912821, 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001252211, 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.109 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679095+0000", 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679096+0000", 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4099999999999998e-07 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679561+0000", 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046529699999999999 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679575+0000", 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3881e-05 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679582+0000", 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7569999999999998e-06 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679627+0000", 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5096000000000003e-05 2024-09-17T07:51:58.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.679633+0000", 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6239999999999999e-06 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.680348+0000", 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071491499999999999 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.405406692, 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010548179999999999, 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:58.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.684602+0000", 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6899999999999997e-07 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685076+0000", 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047355200000000002 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685090+0000", 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4015999999999999e-05 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685096+0000", 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4189999999999999e-06 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685140+0000", 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4027999999999999e-05 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685145+0000", 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2409999999999997e-06 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.685656+0000", 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00051189300000000003 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.6 98/97 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.402568407, 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235564, 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687440+0000", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687441+0000", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.0100000000000004e-07 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687911+0000", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047036099999999998 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687925+0000", 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3910000000000001e-05 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687931+0000", 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687976+0000", 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4972999999999998e-05 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.687980+0000", 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.9829999999999998e-06 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.688675+0000", 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00069505299999999995 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:58.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.398424568, 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010986870000000001, 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691584+0000", 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8200000000000001e-07 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691589+0000", 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.293e-06 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691596+0000", 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.8730000000000001e-06 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691600+0000", 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.0110000000000002e-06 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.691620+0000", 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9803999999999999e-05 2024-09-17T07:51:58.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.692682+0000", 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010623239999999999 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.1 e98/97 cost 1048576)", 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.38418971199999, 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010475269999999999, 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.125 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705819+0000", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7300000000000001e-07 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705826+0000", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9290000000000001e-06 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705834+0000", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6960000000000005e-06 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705838+0000", 2024-09-17T07:51:58.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4549999999999997e-06 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.705860+0000", 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.2243e-05 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:00.706866+0000", 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010057309999999999 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.373735256, 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00169337, 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:58.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716273+0000", 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716274+0000", 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0979999999999999e-06 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716291+0000", 2024-09-17T07:51:58.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6498e-05 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716303+0000", 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2859e-05 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716309+0000", 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5840000000000002e-06 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.716358+0000", 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8631000000000002e-05 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.717966+0000", 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016087 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.132 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.34602588600001, 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00125421, 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743982+0000", 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.743984+0000", 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.0329999999999998e-06 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744019+0000", 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:51:58.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744037+0000", 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7847e-05 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744046+0000", 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.8640000000000002e-06 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.744111+0000", 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.5160999999999995e-05 2024-09-17T07:51:58.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.745237+0000", 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001125633 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.331485464, 2024-09-17T07:51:58.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010969700000000001, 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758523+0000", 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8899999999999999e-07 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758530+0000", 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758539+0000", 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0669999999999998e-06 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758543+0000", 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.3460000000000004e-06 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.758603+0000", 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9812999999999998e-05 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.759620+0000", 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001016552 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.139 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.32775232900001, 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001188415, 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762256+0000", 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762257+0000", 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5370000000000001e-06 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762276+0000", 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8162999999999999e-05 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762300+0000", 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4372999999999999e-05 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762310+0000", 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0193e-05 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.762432+0000", 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000122261 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.763444+0000", 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010118880000000001 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:58.143 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.316710646, 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001332406, 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773298+0000", 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773299+0000", 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5200000000000001e-06 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773319+0000", 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.9483e-05 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773343+0000", 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4162999999999999e-05 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773353+0000", 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0213e-05 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.773462+0000", 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00010932799999999999 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.774630+0000", 2024-09-17T07:51:58.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011676989999999999 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.30233586, 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011100789999999999, 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787672+0000", 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787673+0000", 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2399999999999998e-07 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787682+0000", 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787693+0000", 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.025e-05 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787697+0000", 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6310000000000002e-06 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.787732+0000", 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4288000000000001e-05 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.788783+0000", 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001050905 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.150 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.3 e102/101 cost 1048576)", 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.288379986, 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001108801, 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801628+0000", 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801629+0000", 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.77e-07 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801636+0000", 2024-09-17T07:51:58.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7419999999999997e-06 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801645+0000", 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801650+0000", 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.6140000000000002e-06 2024-09-17T07:51:58.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.801678+0000", 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.8218e-05 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:05.802737+0000", 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001059171 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 72.284745903000001, 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011427939999999999, 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805262+0000", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805263+0000", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7000000000000004e-07 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805274+0000", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1029999999999999e-05 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805285+0000", 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.0764999999999999e-05 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805291+0000", 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7559999999999996e-06 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.805329+0000", 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8784e-05 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.806405+0000", 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010756889999999999 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.6 e123/122 cost 1048576)", 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 72.279740043000004, 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001077555, 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810268+0000", 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810270+0000", 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.539e-06 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810289+0000", 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8889000000000002e-05 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810303+0000", 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4107000000000001e-05 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810309+0000", 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.426e-06 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.810358+0000", 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8372999999999999e-05 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.811346+0000", 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098822100000000006 2024-09-17T07:51:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 72.212459881000001, 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001303954, 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877548+0000", 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877549+0000", 2024-09-17T07:51:58.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1600000000000001e-07 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877553+0000", 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2989999999999998e-06 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877565+0000", 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.1734e-05 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877570+0000", 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0139999999999998e-06 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.877603+0000", 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2540999999999999e-05 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.878852+0000", 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012496499999999999 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.165 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(1.1 123/122 [PushReplyOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head)])", 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 72.203535813000002, 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00109907, 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886472+0000", 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886473+0000", 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4000000000000001e-07 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886477+0000", 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.014e-06 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886489+0000", 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.146e-05 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886493+0000", 2024-09-17T07:51:58.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.886528+0000", 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4740000000000003e-05 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:45.887572+0000", 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001043319 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.170 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:58.171 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T07:51:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-09-17T07:51:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:58.285 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T07:51:58.386 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-09-17T07:51:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-09-17T07:51:58.386 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-09-17T07:51:58.386 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:58.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 45'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@45'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.690585077999998, 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028453100000000002, 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:58.498 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805016+0000", 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805980+0000", 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096373100000000001 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.499 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.805994+0000", 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3627e-05 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806001+0000", 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4329999999999998e-06 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806058+0000", 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6434999999999998e-05 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806062+0000", 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8289999999999998e-06 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.807861+0000", 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001798656 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.501 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.689092055000003, 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029427799999999999, 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.502 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806509+0000", 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.806510+0000", 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2099999999999995e-07 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808008+0000", 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001498465 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808035+0000", 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6723000000000001e-05 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808046+0000", 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1109e-05 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808121+0000", 2024-09-17T07:51:58.504 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4538999999999999e-05 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.808130+0000", 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1460000000000002e-06 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.809452+0000", 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013218769999999999 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.684106471999996, 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022862170000000001, 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811494+0000", 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.811495+0000", 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600000000000005e-07 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812684+0000", 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011887550000000001 2024-09-17T07:51:58.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812697+0000", 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3283e-05 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.508 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812704+0000", 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5860000000000001e-06 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812770+0000", 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6070999999999998e-05 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.812781+0000", 2024-09-17T07:51:58.509 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0798999999999999e-05 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.813781+0000", 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099996700000000004 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.679966949000004, 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028390080000000001, 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.511 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815634+0000", 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.512 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.815635+0000", 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817053+0000", 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00141822 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817062+0000", 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9560000000000003e-06 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817067+0000", 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9139999999999999e-06 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817146+0000", 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9190000000000006e-05 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.817157+0000", 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0959e-05 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.818473+0000", 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001316114 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.675182628000002, 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024796169999999999, 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820418+0000", 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.820419+0000", 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5900000000000002e-07 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821382+0000", 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00096349100000000002 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821395+0000", 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3097e-05 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821401+0000", 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821467+0000", 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5352999999999996e-05 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.821475+0000", 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.456e-06 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.822898+0000", 2024-09-17T07:51:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014226760000000001 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.5 123/122 [PushOp(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 45'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.667411380999994, 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024404259999999999, 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828189+0000", 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.828190+0000", 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829444+0000", 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012540470000000001 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829459+0000", 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.488e-05 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829466+0000", 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5549999999999999e-06 2024-09-17T07:51:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829510+0000", 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4415000000000001e-05 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.829515+0000", 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1200000000000001e-06 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.830630+0000", 2024-09-17T07:51:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114781 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.6 123/122 [PushOp(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head, version: 45'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:6530a684:ns::rbd_data.12c1e94c40fb.000000000000000c:head@45'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.657373140000004, 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026817970000000001, 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.524 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.525 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.838228+0000", 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3200000000000005e-07 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839885+0000", 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016563509999999999 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839898+0000", 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3118999999999999e-05 2024-09-17T07:51:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839904+0000", 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0990000000000004e-06 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839959+0000", 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4716000000000003e-05 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.839963+0000", 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.78e-06 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.840910+0000", 2024-09-17T07:51:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094620000000000001 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head, version: 45'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:4ea51193:ns::rbd_data.12c1e94c40fb.0000000000000000:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.63805558, 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024534919999999998, 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857545+0000", 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.857546+0000", 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8100000000000001e-07 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858518+0000", 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097160599999999997 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858541+0000", 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3285000000000002e-05 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858550+0000", 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1479999999999995e-06 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858601+0000", 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0209e-05 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.858606+0000", 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0150000000000003e-06 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.859999+0000", 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013932479999999999 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.622005716000004, 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028447640000000001, 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873595+0000", 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.873596+0000", 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875190+0000", 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001594624 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875205+0000", 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4450999999999999e-05 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875212+0000", 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1670000000000002e-06 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875277+0000", 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4487999999999996e-05 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875284+0000", 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876440+0000", 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001155536 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.537 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.621422565000003, 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023161689999999999, 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.538 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874178+0000", 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.874179+0000", 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0100000000000005e-07 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875526+0000", 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013475 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.539 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875546+0000", 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9266000000000002e-05 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875556+0000", 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0053e-05 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875617+0000", 2024-09-17T07:51:58.540 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1252000000000002e-05 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.875630+0000", 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3101e-05 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.876494+0000", 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086449599999999997 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.616839222999999, 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023779349999999999, 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:58.542 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.878762+0000", 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6799999999999999e-07 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.543 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880295+0000", 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015324309999999999 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880322+0000", 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6747999999999999e-05 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880336+0000", 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4814000000000001e-05 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880420+0000", 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3535000000000006e-05 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880427+0000", 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8979999999999999e-06 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881140+0000", 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071274100000000003 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.1 123/122 [PushOp(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head, version: 45'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:94c3f284:ns::rbd_data.12c1e94c40fb.000000000000000a:head@45'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.616510172000005, 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024920810000000002, 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.546 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.879091+0000", 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5400000000000001e-07 2024-09-17T07:51:58.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880615+0000", 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001523536 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880627+0000", 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242e-05 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.548 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880637+0000", 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6549999999999999e-06 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880712+0000", 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5069999999999998e-05 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.880720+0000", 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.537e-06 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.881583+0000", 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086320899999999998 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.2 123/122 [PushOp(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head, version: 45'5, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:547e0b44:ns::rbd_data.12c1e94c40fb.000000000000000e:head@45'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 72.608310975999999, 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027465620000000001, 2024-09-17T07:51:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.887290+0000", 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-07 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888593+0000", 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013020989999999999 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888609+0000", 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6339000000000001e-05 2024-09-17T07:51:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888616+0000", 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1130000000000003e-06 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888686+0000", 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9578000000000004e-05 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.888694+0000", 2024-09-17T07:51:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8140000000000007e-06 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:45.890036+0000", 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013421119999999999 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:206 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'261, mlcod=127'258)", 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.953153121, 2024-09-17T07:51:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002984549, 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542448+0000", 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0999999999999998e-07 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542710+0000", 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026201499999999999 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.556 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542722+0000", 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1807999999999999e-05 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542728+0000", 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.874e-06 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542756+0000", 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7657e-05 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.557 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.542769+0000", 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2867999999999999e-05 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545373+0000", 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026044150000000001 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545432+0000", 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9302000000000002e-05 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.558 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head, version: 45'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1a2aa688:::rbd_data.123eb9c2b614.0000000000000009:head@45'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.951626312000002, 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002277378, 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.559 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.543975+0000", 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2799999999999996e-07 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545095+0000", 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001119641 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545112+0000", 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6733999999999999e-05 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545121+0000", 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9089999999999995e-06 2024-09-17T07:51:58.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545207+0000", 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5598000000000003e-05 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.545215+0000", 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0830000000000004e-06 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.546252+0000", 2024-09-17T07:51:58.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010367849999999999 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(1.0 127/126 [PushOp(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head, version: 45'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(1:1e1cf03d:ns::rbd_data.12c1e94c40fb.0000000000000005:head@45'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.920381305999996, 2024-09-17T07:51:58.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002310416, 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.575220+0000", 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6099999999999999e-07 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576206+0000", 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000985545 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576232+0000", 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5542e-05 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576244+0000", 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2017e-05 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576319+0000", 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5318e-05 2024-09-17T07:51:58.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.576329+0000", 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0564000000000001e-05 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.577530+0000", 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012005690000000001 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.567 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.5371.0:276 3.3 e127/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:head v 127'270, mlcod=127'268)", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 68.868558246999996, 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027703630000000001, 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627043+0000", 2024-09-17T07:51:58.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627044+0000", 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0800000000000003e-07 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627889+0000", 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084559999999999995 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627906+0000", 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6818000000000001e-05 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627914+0000", 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.892e-06 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627958+0000", 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3727999999999999e-05 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.627979+0000", 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1107999999999998e-05 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629770+0000", 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017912570000000001 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:50:49.629813+0000", 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3052e-05 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.6934.0:68 3.3 e163/124 3:e63daa6a:::rbd_object_map.123eb9c2b614:head v 163'441, mlcod=161'438)", 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:53.714180+0000", 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7814210570000002, 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032571499999999999, 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714180+0000", 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714180+0000", 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714180+0000", 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9399999999999999e-07 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714186+0000", 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5589999999999996e-06 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714194+0000", 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8119999999999997e-06 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714200+0000", 2024-09-17T07:51:58.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4739999999999996e-06 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714243+0000", 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2895999999999997e-05 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714265+0000", 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2291999999999999e-05 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:58.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.717410+0000", 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031448650000000002 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.717437+0000", 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7058000000000001e-05 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.4.0:456 3.3 e163/124 3:c954808a:::rbd_data.123eb9c2b614.0000000000000000:6 v 163'443, mlcod=161'438)", 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:53.714750+0000", 2024-09-17T07:51:58.576 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7808506499999996, 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027021430000000002, 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714750+0000", 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714750+0000", 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714751+0000", 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4499999999999997e-07 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714755+0000", 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714765+0000", 2024-09-17T07:51:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7559999999999994e-06 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714770+0000", 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9840000000000001e-06 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714803+0000", 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.273e-05 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.714821+0000", 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8461000000000002e-05 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.717441+0000", 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026194569999999999 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.717452+0000", 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1556000000000001e-05 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.4.0:460 3.3 e163/124 3:dd6152af:::rbd_data.123eb9c2b614.000000000000000d:7 v 163'451, mlcod=163'447)", 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-09-17T07:51:53.722190+0000", 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7734111400000003, 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028232930000000002, 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722190+0000", 2024-09-17T07:51:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722190+0000", 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722190+0000", 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8800000000000003e-07 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722196+0000", 2024-09-17T07:51:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9460000000000003e-06 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722204+0000", 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2740000000000007e-06 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722208+0000", 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3330000000000001e-06 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722240+0000", 2024-09-17T07:51:58.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2147000000000002e-05 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.722252+0000", 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2004999999999999e-05 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.724985+0000", 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027328529999999999 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-09-17T07:51:53.725013+0000", 2024-09-17T07:51:58.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8246999999999999e-05 2024-09-17T07:51:58.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-17T07:51:58.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-17T07:51:58.585 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:51:58.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T07:51:58.689 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-09-17T07:51:58.689 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:58.698 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T07:51:58.710 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.2 pg_upmap mapping to [3,1] 2024-09-17T07:51:58.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T07:51:58.795 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:58.796 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-09-17T07:51:58.796 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-09-17T07:51:58.796 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-09-17T07:51:58.796 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:58.804 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:58.904 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.386417379999997, 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001374019, 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:58.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516613+0000", 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8800000000000002e-07 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516623+0000", 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.4239999999999999e-06 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516632+0000", 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.3950000000000007e-06 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516636+0000", 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1219999999999997e-06 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.516670+0000", 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.3247000000000002e-05 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.517987+0000", 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013172430000000001 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.370797714000005, 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001153928, 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532232+0000", 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532233+0000", 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9800000000000004e-07 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532240+0000", 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532249+0000", 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:51:58.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532253+0000", 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4689999999999999e-06 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.532323+0000", 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.9621000000000003e-05 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.533386+0000", 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010636910000000001 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPull(1.0 e127/126 cost 1048576)", 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.354511540000004, 2024-09-17T07:51:58.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012482940000000001, 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548519+0000", 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.3499999999999995e-07 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548532+0000", 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2847000000000001e-05 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548540+0000", 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5830000000000001e-06 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548545+0000", 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.5609999999999998e-06 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.548582+0000", 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.6526999999999998e-05 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:49.549767+0000", 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001185041 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.384601259999997, 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027517140000000002, 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.916 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.518429+0000", 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2600000000000002e-07 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519840+0000", 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014109929999999999 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519858+0000", 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7337e-05 2024-09-17T07:51:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519866+0000", 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6890000000000004e-06 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519937+0000", 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1857e-05 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.519945+0000", 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0949999999999996e-06 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.521181+0000", 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001235117 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.920 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.379892401999996, 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021277900000000001, 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523138+0000", 2024-09-17T07:51:58.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.523139+0000", 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.7700000000000003e-07 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524144+0000", 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001005426 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524162+0000", 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7859999999999998e-05 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524171+0000", 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.1290000000000002e-06 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524236+0000", 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4610999999999995e-05 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.524242+0000", 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.525266+0000", 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010231260000000001 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:58.924 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.375995349999997, 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0019829299999999999, 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:58.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527035+0000", 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6599999999999996e-07 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527982+0000", 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094708399999999995 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.527998+0000", 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.5201999999999999e-05 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528004+0000", 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.4919999999999998e-06 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528061+0000", 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.7278999999999998e-05 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.528068+0000", 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.097e-06 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.529018+0000", 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094921000000000003 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head, version: 53'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:head@53'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.372406174000005, 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0022067810000000001, 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:58.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.530624+0000", 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.4099999999999999e-07 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531621+0000", 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099604300000000006 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531634+0000", 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3689000000000001e-05 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531641+0000", 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531701+0000", 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9456999999999997e-05 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.531716+0000", 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.502e-05 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.532831+0000", 2024-09-17T07:51:58.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011147710000000001 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.367921856999999, 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020280900000000002, 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535108+0000", 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.535109+0000", 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.6499999999999998e-07 2024-09-17T07:51:58.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536043+0000", 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093433700000000004 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536060+0000", 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7113e-05 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536069+0000", 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0220000000000004e-06 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536128+0000", 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9713999999999998e-05 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.536136+0000", 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.8569999999999995e-06 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.537136+0000", 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010002819999999999 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:58.937 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.363989826999997, 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0021243740000000001, 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539040+0000", 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539041+0000", 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7499999999999999e-07 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.539999+0000", 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095760099999999998 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540016+0000", 2024-09-17T07:51:58.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7274000000000001e-05 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540023+0000", 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.2620000000000001e-06 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540083+0000", 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0102999999999997e-05 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.540092+0000", 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.3559999999999993e-06 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.541165+0000", 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0010730030000000001 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.941 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.360202987999998, 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020105570000000001, 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542827+0000", 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.542828+0000", 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.1500000000000005e-07 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543816+0000", 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098795400000000009 2024-09-17T07:51:58.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543829+0000", 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3721e-05 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543837+0000", 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4540000000000001e-06 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543898+0000", 2024-09-17T07:51:58.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.1359999999999995e-05 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.543904+0000", 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.6149999999999996e-06 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.544838+0000", 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00093393800000000004 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head, version: 53'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:head@53'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.356551306, 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020839169999999998, 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.546479+0000", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.89e-07 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547440+0000", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00096050300000000003 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547454+0000", 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.417e-05 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547460+0000", 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.0850000000000002e-06 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547542+0000", 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1767000000000005e-05 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.547546+0000", 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4730000000000002e-06 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.548563+0000", 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00101643 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:51:58.949 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.352442865, 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020214439999999998, 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550587+0000", 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.550589+0000", 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.61e-06 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551535+0000", 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094618399999999998 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551551+0000", 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6087e-05 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551578+0000", 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.6367e-05 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551660+0000", 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.1990999999999995e-05 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.551664+0000", 2024-09-17T07:51:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.9189999999999998e-06 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.552609+0000", 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00094428600000000002 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.348842794000007, 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0016834440000000001, 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554187+0000", 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.554188+0000", 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0800000000000004e-07 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555141+0000", 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00095264999999999998 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555158+0000", 2024-09-17T07:51:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.6872e-05 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555165+0000", 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.1960000000000002e-06 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555209+0000", 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4486000000000001e-05 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555214+0000", 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7539999999999997e-06 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.555871+0000", 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00065667799999999995 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.345476200999997, 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017156980000000001, 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.557554+0000", 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.0900000000000002e-07 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558536+0000", 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098119500000000003 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558548+0000", 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2745000000000001e-05 2024-09-17T07:51:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558574+0000", 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5401e-05 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558654+0000", 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0320000000000003e-05 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.558659+0000", 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.4309999999999996e-06 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.559270+0000", 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00061109700000000001 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(3.1 128/127 [PushOp(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head, version: 53'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:head@53'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.342084467000006, 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0018243560000000001, 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.560946+0000", 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.7800000000000002e-07 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561931+0000", 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00098499099999999995 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561945+0000", 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3624e-05 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.561951+0000", 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.2129999999999996e-06 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562010+0000", 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8702999999999997e-05 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562014+0000", 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.5129999999999998e-06 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:50:50.562770+0000", 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00075583400000000004 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.2.0:77 3.1 e163/127 3:a10dfb9f:::rbd_data.123eb9c2b614.0000000000000007:7 v 163'106, mlcod=162'102)", 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2078302790000004, 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029160549999999999, 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695200+0000", 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.97e-07 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695204+0000", 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.8959999999999996e-06 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695211+0000", 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.5560000000000002e-06 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695217+0000", 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.203e-06 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695259+0000", 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2700000000000001e-05 2024-09-17T07:51:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695284+0000", 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4643000000000001e-05 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698090+0000", 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028058620000000001 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698116+0000", 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5998000000000001e-05 2024-09-17T07:51:58.969 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:468 3.2 e163/154 3:4272337b:::rbd_data.123eb9c2b614.0000000000000001:7 v 163'63, mlcod=162'60)", 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2076445419999997, 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0034243440000000002, 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695386+0000", 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4999999999999999e-07 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695392+0000", 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.8080000000000001e-06 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695401+0000", 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.9809999999999992e-06 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695406+0000", 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.631e-06 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695441+0000", 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.4848000000000002e-05 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.695460+0000", 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.8553e-05 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698778+0000", 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0033180190000000002 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698810+0000", 2024-09-17T07:51:58.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.2254e-05 2024-09-17T07:51:58.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.2.0:78 3.1 e163/127 3:a9a0ba02:::rbd_data.123eb9c2b614.000000000000000a:7 v 163'108, mlcod=162'102)", 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.696589+0000", 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2064407179999996, 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017333170000000001, 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696589+0000", 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696589+0000", 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696590+0000", 2024-09-17T07:51:58.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 6.6499999999999999e-07 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696596+0000", 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9800000000000003e-06 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696604+0000", 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.7100000000000007e-06 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.696608+0000", 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.1119999999999998e-06 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697020+0000", 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00041253200000000001 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697033+0000", 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.2461e-05 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698299+0000", 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00126636 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698323+0000", 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3496999999999999e-05 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.1.0:470 3.2 e163/154 3:7e401a81:::rbd_data.123eb9c2b614.000000000000000f:7 v 163'65, mlcod=162'60)", 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.697259+0000", 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2057711739999997, 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001703244, 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697259+0000", 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697259+0000", 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697260+0000", 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.0599999999999999e-07 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697266+0000", 2024-09-17T07:51:58.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 5.9370000000000001e-06 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697280+0000", 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.4283e-05 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697287+0000", 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 7.4769999999999997e-06 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697323+0000", 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 3.5281000000000003e-05 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.697341+0000", 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.7841000000000001e-05 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698939+0000", 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001598066 2024-09-17T07:51:58.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.698962+0000", 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.3553e-05 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(osd.2.0:79 3.1 e163/127 3:a333b343:::rbd_data.123eb9c2b614.0000000000000003:7 v 163'110, mlcod=163'106)", 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 5.2036225920000003, 2024-09-17T07:51:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001263007, 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699408+0000", 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.71e-07 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699423+0000", 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.446e-05 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699431+0000", 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4740000000000005e-06 2024-09-17T07:51:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699436+0000", 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4.2710000000000003e-06 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699460+0000", 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.4692999999999999e-05 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.699474+0000", 2024-09-17T07:51:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 1.3528e-05 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.700645+0000", 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0011708000000000001 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-09-17T07:51:53.700671+0000", 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 2.5910000000000001e-05 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-09-17T07:51:58.990 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-17T07:51:58.990 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-17T07:51:59.715 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-09-17T07:51:59.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T07:52:00.717 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-09-17T07:52:03.725 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-17T07:52:03.725 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-17T07:52:03.726 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-09-17T07:52:04.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-09-17T07:52:04.019+0000 7ffb533f1640 -1 mon.a@0(leader).osd e170 definitely_dead 0 2024-09-17T07:52:04.728 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.5. 2024-09-17T07:52:04.746 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-17T07:52:04.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-17T07:52:04.862 INFO:teuthology.orchestra.run.smithi026.stdout:mon.a: {} 2024-09-17T07:52:04.863 INFO:teuthology.orchestra.run.smithi026.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:52:04.872 INFO:teuthology.orchestra.run.smithi026.stdout:mon.b: {} 2024-09-17T07:52:04.872 INFO:teuthology.orchestra.run.smithi026.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:52:04.880 INFO:teuthology.orchestra.run.smithi026.stdout:mon.c: {} 2024-09-17T07:52:04.880 INFO:teuthology.orchestra.run.smithi026.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:52:04.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-17T07:52:05.736 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:52:05.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-17T07:52:06.737 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-17T07:52:06.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-17T07:52:07.742 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:52:07.758 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-17T07:52:08.746 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-17T07:52:08.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-17T07:52:09.751 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:52:09.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-17T07:52:10.756 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-17T07:52:10.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-17T07:52:11.762 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:52:11.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-17T07:52:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-17T07:52:12.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-17T07:52:13.772 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:52:13.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-17T07:52:14.777 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-17T07:52:14.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-17T07:52:15.782 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T07:52:15.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-17T07:52:16.787 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-17T07:52:16.804 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-17T07:52:16.805 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-17T07:52:16.805 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-17T07:52:16.805 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-17T07:52:16.805 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T07:52:16.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:52:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:17.084 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":183,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:52:16.783369+0000","last_up_change":"2024-09-17T07:50:48.503993+0000","last_in_change":"2024-09-17T07:52:04.724383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":38,"up_from":121,"up_thru":154,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":69763},{"type":"v1","addr":"172.21.15.26:6801","nonce":69763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":69763},{"type":"v1","addr":"172.21.15.26:6803","nonce":69763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":69763},{"type":"v1","addr":"172.21.15.26:6807","nonce":69763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":69763},{"type":"v1","addr":"172.21.15.26:6805","nonce":69763}]},"public_addr":"172.21.15.26:6801/69763","cluster_addr":"172.21.15.26:6803/69763","heartbeat_back_addr":"172.21.15.26:6807/69763","heartbeat_front_addr":"172.21.15.26:6805/69763","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":49,"up_from":126,"up_thru":39,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":61678},{"type":"v1","addr":"172.21.15.94:6817","nonce":61678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":61678},{"type":"v1","addr":"172.21.15.94:6819","nonce":61678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":61678},{"type":"v1","addr":"172.21.15.94:6823","nonce":61678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":61678},{"type":"v1","addr":"172.21.15.94:6821","nonce":61678}]},"public_addr":"172.21.15.94:6817/61678","cluster_addr":"172.21.15.94:6819/61678","heartbeat_back_addr":"172.21.15.94:6823/61678","heartbeat_front_addr":"172.21.15.94:6821/61678","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":68,"up_from":96,"up_thru":171,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":57427},{"type":"v1","addr":"172.21.15.94:6801","nonce":57427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":57427},{"type":"v1","addr":"172.21.15.94:6803","nonce":57427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":57427},{"type":"v1","addr":"172.21.15.94:6807","nonce":57427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":57427},{"type":"v1","addr":"172.21.15.94:6805","nonce":57427}]},"public_addr":"172.21.15.94:6801/57427","cluster_addr":"172.21.15.94:6803/57427","heartbeat_back_addr":"172.21.15.94:6807/57427","heartbeat_front_addr":"172.21.15.94:6805/57427","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":50},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":69}],"pg_upmap":[{"pgid":"1.2","osds":[3,1]},{"pgid":"1.5","osds":[1,0]},{"pgid":"3.1","osds":[3,2]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:52:17.096 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T07:52:17.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:52:17.369 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:17.370 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":183,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:52:16.783369+0000","last_up_change":"2024-09-17T07:50:48.503993+0000","last_in_change":"2024-09-17T07:52:04.724383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":38,"up_from":121,"up_thru":154,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":69763},{"type":"v1","addr":"172.21.15.26:6801","nonce":69763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":69763},{"type":"v1","addr":"172.21.15.26:6803","nonce":69763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":69763},{"type":"v1","addr":"172.21.15.26:6807","nonce":69763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":69763},{"type":"v1","addr":"172.21.15.26:6805","nonce":69763}]},"public_addr":"172.21.15.26:6801/69763","cluster_addr":"172.21.15.26:6803/69763","heartbeat_back_addr":"172.21.15.26:6807/69763","heartbeat_front_addr":"172.21.15.26:6805/69763","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":49,"up_from":126,"up_thru":39,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":61678},{"type":"v1","addr":"172.21.15.94:6817","nonce":61678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":61678},{"type":"v1","addr":"172.21.15.94:6819","nonce":61678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":61678},{"type":"v1","addr":"172.21.15.94:6823","nonce":61678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":61678},{"type":"v1","addr":"172.21.15.94:6821","nonce":61678}]},"public_addr":"172.21.15.94:6817/61678","cluster_addr":"172.21.15.94:6819/61678","heartbeat_back_addr":"172.21.15.94:6823/61678","heartbeat_front_addr":"172.21.15.94:6821/61678","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":68,"up_from":96,"up_thru":171,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":57427},{"type":"v1","addr":"172.21.15.94:6801","nonce":57427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":57427},{"type":"v1","addr":"172.21.15.94:6803","nonce":57427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":57427},{"type":"v1","addr":"172.21.15.94:6807","nonce":57427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":57427},{"type":"v1","addr":"172.21.15.94:6805","nonce":57427}]},"public_addr":"172.21.15.94:6801/57427","cluster_addr":"172.21.15.94:6803/57427","heartbeat_back_addr":"172.21.15.94:6807/57427","heartbeat_front_addr":"172.21.15.94:6805/57427","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":50},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":69}],"pg_upmap":[{"pgid":"1.2","osds":[3,1]},{"pgid":"1.5","osds":[1,0]},{"pgid":"3.1","osds":[3,2]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:52:17.382 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T07:52:17.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T07:52:17.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T07:52:17.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T07:52:17.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T07:52:17.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T07:52:17.536 INFO:teuthology.orchestra.run.smithi026.stdout:47244640325 2024-09-17T07:52:17.537 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T07:52:17.538 INFO:teuthology.orchestra.run.smithi026.stdout:412316860446 2024-09-17T07:52:17.538 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T07:52:17.581 INFO:teuthology.orchestra.run.smithi026.stdout:47244640326 2024-09-17T07:52:17.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T07:52:17.582 INFO:teuthology.orchestra.run.smithi026.stdout:519691042836 2024-09-17T07:52:17.582 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T07:52:17.591 INFO:teuthology.orchestra.run.smithi026.stdout:47244640325 2024-09-17T07:52:17.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T07:52:17.593 INFO:teuthology.orchestra.run.smithi026.stdout:541165879316 2024-09-17T07:52:17.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T07:52:17.931 INFO:teuthology.orchestra.run.smithi026.stdout:519691042835 2024-09-17T07:52:17.945 INFO:tasks.ceph.ceph_manager.ceph:need seq 519691042836 got 519691042835 for osd.0 2024-09-17T07:52:17.960 INFO:teuthology.orchestra.run.smithi026.stdout:47244640324 2024-09-17T07:52:17.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640325 got 47244640324 for osd.4 2024-09-17T07:52:18.023 INFO:teuthology.orchestra.run.smithi026.stdout:47244640325 2024-09-17T07:52:18.036 INFO:teuthology.orchestra.run.smithi026.stdout:412316860445 2024-09-17T07:52:18.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640326 got 47244640325 for osd.1 2024-09-17T07:52:18.045 INFO:teuthology.orchestra.run.smithi026.stdout:47244640324 2024-09-17T07:52:18.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 412316860446 got 412316860445 for osd.5 2024-09-17T07:52:18.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640325 got 47244640324 for osd.2 2024-09-17T07:52:18.085 INFO:teuthology.orchestra.run.smithi026.stdout:541165879314 2024-09-17T07:52:18.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 541165879316 got 541165879314 for osd.3 2024-09-17T07:52:18.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T07:52:18.973 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T07:52:19.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T07:52:19.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T07:52:19.059 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T07:52:19.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T07:52:19.402 INFO:teuthology.orchestra.run.smithi026.stdout:519691042835 2024-09-17T07:52:19.421 INFO:tasks.ceph.ceph_manager.ceph:need seq 519691042836 got 519691042835 for osd.0 2024-09-17T07:52:19.435 INFO:teuthology.orchestra.run.smithi026.stdout:47244640325 2024-09-17T07:52:19.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640326 got 47244640325 for osd.1 2024-09-17T07:52:19.452 INFO:teuthology.orchestra.run.smithi026.stdout:47244640324 2024-09-17T07:52:19.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640325 got 47244640324 for osd.4 2024-09-17T07:52:19.478 INFO:teuthology.orchestra.run.smithi026.stdout:47244640324 2024-09-17T07:52:19.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640325 got 47244640324 for osd.2 2024-09-17T07:52:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:412316860445 2024-09-17T07:52:19.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 412316860446 got 412316860445 for osd.5 2024-09-17T07:52:19.546 INFO:teuthology.orchestra.run.smithi026.stdout:541165879314 2024-09-17T07:52:19.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 541165879316 got 541165879314 for osd.3 2024-09-17T07:52:20.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T07:52:20.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T07:52:20.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T07:52:20.492 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T07:52:20.522 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T07:52:20.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T07:52:20.790 INFO:teuthology.orchestra.run.smithi026.stdout:47244640326 2024-09-17T07:52:20.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640326 got 47244640326 for osd.1 2024-09-17T07:52:20.804 DEBUG:teuthology.parallel:result is None 2024-09-17T07:52:20.890 INFO:teuthology.orchestra.run.smithi026.stdout:541165879316 2024-09-17T07:52:20.907 INFO:tasks.ceph.ceph_manager.ceph:need seq 541165879316 got 541165879316 for osd.3 2024-09-17T07:52:20.907 DEBUG:teuthology.parallel:result is None 2024-09-17T07:52:20.908 INFO:teuthology.orchestra.run.smithi026.stdout:47244640325 2024-09-17T07:52:20.908 INFO:teuthology.orchestra.run.smithi026.stdout:519691042837 2024-09-17T07:52:20.921 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640325 got 47244640325 for osd.2 2024-09-17T07:52:20.921 DEBUG:teuthology.parallel:result is None 2024-09-17T07:52:20.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 519691042836 got 519691042837 for osd.0 2024-09-17T07:52:20.926 DEBUG:teuthology.parallel:result is None 2024-09-17T07:52:20.944 INFO:teuthology.orchestra.run.smithi026.stdout:47244640325 2024-09-17T07:52:20.952 INFO:teuthology.orchestra.run.smithi026.stdout:412316860447 2024-09-17T07:52:20.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640325 got 47244640325 for osd.4 2024-09-17T07:52:20.957 DEBUG:teuthology.parallel:result is None 2024-09-17T07:52:20.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 412316860446 got 412316860447 for osd.5 2024-09-17T07:52:20.965 DEBUG:teuthology.parallel:result is None 2024-09-17T07:52:20.965 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:52:20.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:21.276 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:21.287 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-17T07:52:19.474280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981144,"kb_used_data":169908,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249896,"statfs":{"total":579820584960,"available":577791893504,"internally_reserved":0,"allocated":173985792,"data_stored":166337904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.698674"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":107,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786860+0000","last_change":"2024-09-17T07:52:16.787095+0000","last_active":"2024-09-17T07:52:16.786819+0000","last_peered":"2024-09-17T07:51:55.701674+0000","last_clean":"2024-09-17T07:51:55.701674+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:52:16.786860+0000","last_undegraded":"2024-09-17T07:52:16.786860+0000","last_fullsized":"2024-09-17T07:52:16.786860+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:58:51.723619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":900,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.781969+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:15.781969+0000","last_peered":"2024-09-17T07:52:15.781969+0000","last_clean":"2024-09-17T07:52:15.781969+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:15.781969+0000","last_undegraded":"2024-09-17T07:52:15.781969+0000","last_fullsized":"2024-09-17T07:52:15.781969+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2212,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-17T07:52:14.926578+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:14.926578+0000","last_peered":"2024-09-17T07:52:14.926578+0000","last_clean":"2024-09-17T07:52:14.926578+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:14.926578+0000","last_undegraded":"2024-09-17T07:52:14.926578+0000","last_fullsized":"2024-09-17T07:52:14.926578+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":5,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786779+0000","last_change":"2024-09-17T07:52:16.787004+0000","last_active":"2024-09-17T07:51:59.714631+0000","last_peered":"2024-09-17T07:51:55.701133+0000","last_clean":"2024-09-17T07:51:55.701133+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:52:16.786779+0000","last_undegraded":"2024-09-17T07:52:16.786779+0000","last_fullsized":"2024-09-17T07:52:16.786779+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:27.731564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782275+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:15.782275+0000","last_peered":"2024-09-17T07:52:15.782275+0000","last_clean":"2024-09-17T07:52:15.782275+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:15.782275+0000","last_undegraded":"2024-09-17T07:52:15.782275+0000","last_fullsized":"2024-09-17T07:52:15.782275+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":364,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.781944+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:15.781944+0000","last_peered":"2024-09-17T07:52:15.781944+0000","last_clean":"2024-09-17T07:52:15.781944+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:15.781944+0000","last_undegraded":"2024-09-17T07:52:15.781944+0000","last_fullsized":"2024-09-17T07:52:15.781944+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":101,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786702+0000","last_change":"2024-09-17T07:52:16.786930+0000","last_active":"2024-09-17T07:50:47.515749+0000","last_peered":"2024-09-17T07:46:53.908455+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:52:16.786702+0000","last_undegraded":"2024-09-17T07:52:16.786702+0000","last_fullsized":"2024-09-17T07:52:16.786702+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:59.966450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":401,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786901+0000","last_change":"2024-09-17T07:52:16.787057+0000","last_active":"2024-09-17T07:52:16.786870+0000","last_peered":"2024-09-17T07:51:55.828289+0000","last_clean":"2024-09-17T07:51:55.828289+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:52:16.786901+0000","last_undegraded":"2024-09-17T07:52:16.786901+0000","last_fullsized":"2024-09-17T07:52:16.786901+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:19.570284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782138+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:15.782138+0000","last_peered":"2024-09-17T07:52:15.782138+0000","last_clean":"2024-09-17T07:52:15.782138+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:15.782138+0000","last_undegraded":"2024-09-17T07:52:15.782138+0000","last_fullsized":"2024-09-17T07:52:15.782138+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":218,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782024+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:15.782024+0000","last_peered":"2024-09-17T07:52:15.782024+0000","last_clean":"2024-09-17T07:52:15.782024+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:15.782024+0000","last_undegraded":"2024-09-17T07:52:15.782024+0000","last_fullsized":"2024-09-17T07:52:15.782024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":98,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786821+0000","last_change":"2024-09-17T07:52:16.787033+0000","last_active":"2024-09-17T07:52:16.786793+0000","last_peered":"2024-09-17T07:51:55.702144+0000","last_clean":"2024-09-17T07:51:55.702144+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:52:16.786821+0000","last_undegraded":"2024-09-17T07:52:16.786821+0000","last_fullsized":"2024-09-17T07:52:16.786821+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:20.762102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337176,"kb_used_data":39052,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034664,"statfs":{"total":96636764160,"available":96291495936,"internally_reserved":0,"allocated":39989248,"data_stored":38708546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.505},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.46400000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.379}]}]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307832,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064008,"statfs":{"total":96636764160,"available":96321544192,"internally_reserved":0,"allocated":2207744,"data_stored":959483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.624}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.65400000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.49099999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.42899999999999999}]}]},{"osd":3,"up_from":126,"seq":541165879316,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329384,"kb_used_data":31444,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042456,"statfs":{"total":96636764160,"available":96299474944,"internally_reserved":0,"allocated":32198656,"data_stored":30915289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.59499999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.625}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.48199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320156,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051684,"statfs":{"total":96636764160,"available":96308924416,"internally_reserved":0,"allocated":14819328,"data_stored":13547671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.247},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.35399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.432}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38900000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640326,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361704,"kb_used_data":56020,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010136,"statfs":{"total":96636764160,"available":96266379264,"internally_reserved":0,"allocated":57364480,"data_stored":56076336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.253}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.311}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54500000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.47699999999999998}]}]},{"osd":0,"up_from":121,"seq":519691042837,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324892,"kb_used_data":26764,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046948,"statfs":{"total":96636764160,"available":96304074752,"internally_reserved":0,"allocated":27406336,"data_stored":26130579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.45000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.623}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:21.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:21.559 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:21.571 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-17T07:52:21.474774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981144,"kb_used_data":169908,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249896,"statfs":{"total":579820584960,"available":577791893504,"internally_reserved":0,"allocated":173985792,"data_stored":166337904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001210"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":107,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786860+0000","last_change":"2024-09-17T07:52:16.787095+0000","last_active":"2024-09-17T07:52:16.786819+0000","last_peered":"2024-09-17T07:51:55.701674+0000","last_clean":"2024-09-17T07:51:55.701674+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:52:16.786860+0000","last_undegraded":"2024-09-17T07:52:16.786860+0000","last_fullsized":"2024-09-17T07:52:16.786860+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:58:51.723619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":900,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.781969+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:15.781969+0000","last_peered":"2024-09-17T07:52:15.781969+0000","last_clean":"2024-09-17T07:52:15.781969+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:15.781969+0000","last_undegraded":"2024-09-17T07:52:15.781969+0000","last_fullsized":"2024-09-17T07:52:15.781969+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2212,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-17T07:52:14.926578+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:14.926578+0000","last_peered":"2024-09-17T07:52:14.926578+0000","last_clean":"2024-09-17T07:52:14.926578+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:14.926578+0000","last_undegraded":"2024-09-17T07:52:14.926578+0000","last_fullsized":"2024-09-17T07:52:14.926578+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":5,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786779+0000","last_change":"2024-09-17T07:52:16.787004+0000","last_active":"2024-09-17T07:51:59.714631+0000","last_peered":"2024-09-17T07:51:55.701133+0000","last_clean":"2024-09-17T07:51:55.701133+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:52:16.786779+0000","last_undegraded":"2024-09-17T07:52:16.786779+0000","last_fullsized":"2024-09-17T07:52:16.786779+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:27.731564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782275+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:15.782275+0000","last_peered":"2024-09-17T07:52:15.782275+0000","last_clean":"2024-09-17T07:52:15.782275+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:15.782275+0000","last_undegraded":"2024-09-17T07:52:15.782275+0000","last_fullsized":"2024-09-17T07:52:15.782275+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":364,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.781944+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:15.781944+0000","last_peered":"2024-09-17T07:52:15.781944+0000","last_clean":"2024-09-17T07:52:15.781944+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:15.781944+0000","last_undegraded":"2024-09-17T07:52:15.781944+0000","last_fullsized":"2024-09-17T07:52:15.781944+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":101,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786702+0000","last_change":"2024-09-17T07:52:16.786930+0000","last_active":"2024-09-17T07:50:47.515749+0000","last_peered":"2024-09-17T07:46:53.908455+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:52:16.786702+0000","last_undegraded":"2024-09-17T07:52:16.786702+0000","last_fullsized":"2024-09-17T07:52:16.786702+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:59.966450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":401,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786901+0000","last_change":"2024-09-17T07:52:16.787057+0000","last_active":"2024-09-17T07:52:16.786870+0000","last_peered":"2024-09-17T07:51:55.828289+0000","last_clean":"2024-09-17T07:51:55.828289+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:52:16.786901+0000","last_undegraded":"2024-09-17T07:52:16.786901+0000","last_fullsized":"2024-09-17T07:52:16.786901+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:19.570284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782138+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:15.782138+0000","last_peered":"2024-09-17T07:52:15.782138+0000","last_clean":"2024-09-17T07:52:15.782138+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:15.782138+0000","last_undegraded":"2024-09-17T07:52:15.782138+0000","last_fullsized":"2024-09-17T07:52:15.782138+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":218,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782024+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:15.782024+0000","last_peered":"2024-09-17T07:52:15.782024+0000","last_clean":"2024-09-17T07:52:15.782024+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:15.782024+0000","last_undegraded":"2024-09-17T07:52:15.782024+0000","last_fullsized":"2024-09-17T07:52:15.782024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":98,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786821+0000","last_change":"2024-09-17T07:52:16.787033+0000","last_active":"2024-09-17T07:52:16.786793+0000","last_peered":"2024-09-17T07:51:55.702144+0000","last_clean":"2024-09-17T07:51:55.702144+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:52:16.786821+0000","last_undegraded":"2024-09-17T07:52:16.786821+0000","last_fullsized":"2024-09-17T07:52:16.786821+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:20.762102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337176,"kb_used_data":39052,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034664,"statfs":{"total":96636764160,"available":96291495936,"internally_reserved":0,"allocated":39989248,"data_stored":38708546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.505},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.46400000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.379}]}]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307832,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064008,"statfs":{"total":96636764160,"available":96321544192,"internally_reserved":0,"allocated":2207744,"data_stored":959483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.624}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.65400000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.49099999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.42899999999999999}]}]},{"osd":3,"up_from":126,"seq":541165879316,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329384,"kb_used_data":31444,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042456,"statfs":{"total":96636764160,"available":96299474944,"internally_reserved":0,"allocated":32198656,"data_stored":30915289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.59499999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.625}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.48199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320156,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051684,"statfs":{"total":96636764160,"available":96308924416,"internally_reserved":0,"allocated":14819328,"data_stored":13547671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.247},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.35399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.432}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38900000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640326,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361704,"kb_used_data":56020,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010136,"statfs":{"total":96636764160,"available":96266379264,"internally_reserved":0,"allocated":57364480,"data_stored":56076336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.253}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.311}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54500000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.47699999999999998}]}]},{"osd":0,"up_from":121,"seq":519691042837,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324892,"kb_used_data":26764,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046948,"statfs":{"total":96636764160,"available":96304074752,"internally_reserved":0,"allocated":27406336,"data_stored":26130579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.45000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.623}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:21.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T07:52:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:21.908 INFO:teuthology.orchestra.run.smithi026.stdout:{"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive, 1 pg peering","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":340,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":184,"num_osds":6,"num_up_osds":6,"osd_up_since":1726559448,"num_in_osds":6,"osd_in_since":1726559524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"peering","count":5}],"num_pgs":13,"num_pools":3,"num_objects":96,"data_bytes":80283279,"bytes_used":2028691456,"bytes_avail":577791893504,"bytes_total":579820584960,"inactive_pgs_ratio":0.38461539149284363},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-17T07:51:49.468866+0000","services":{}},"progress_events":{}} 2024-09-17T07:52:21.920 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T07:52:21.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:22.189 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:22.189 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:22.200 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-17T07:52:21.474774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981144,"kb_used_data":169908,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249896,"statfs":{"total":579820584960,"available":577791893504,"internally_reserved":0,"allocated":173985792,"data_stored":166337904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001210"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":107,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786860+0000","last_change":"2024-09-17T07:52:16.787095+0000","last_active":"2024-09-17T07:52:16.786819+0000","last_peered":"2024-09-17T07:51:55.701674+0000","last_clean":"2024-09-17T07:51:55.701674+0000","last_became_active":"2024-09-17T07:50:49.510801+0000","last_became_peered":"2024-09-17T07:50:49.510801+0000","last_unstale":"2024-09-17T07:52:16.786860+0000","last_undegraded":"2024-09-17T07:52:16.786860+0000","last_fullsized":"2024-09-17T07:52:16.786860+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:58:51.723619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":900,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.781969+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:15.781969+0000","last_peered":"2024-09-17T07:52:15.781969+0000","last_clean":"2024-09-17T07:52:15.781969+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:15.781969+0000","last_undegraded":"2024-09-17T07:52:15.781969+0000","last_fullsized":"2024-09-17T07:52:15.781969+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2212,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-17T07:52:14.926578+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:14.926578+0000","last_peered":"2024-09-17T07:52:14.926578+0000","last_clean":"2024-09-17T07:52:14.926578+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:14.926578+0000","last_undegraded":"2024-09-17T07:52:14.926578+0000","last_fullsized":"2024-09-17T07:52:14.926578+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":5,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786779+0000","last_change":"2024-09-17T07:52:16.787004+0000","last_active":"2024-09-17T07:51:59.714631+0000","last_peered":"2024-09-17T07:51:55.701133+0000","last_clean":"2024-09-17T07:51:55.701133+0000","last_became_active":"2024-09-17T07:50:53.529559+0000","last_became_peered":"2024-09-17T07:50:53.529559+0000","last_unstale":"2024-09-17T07:52:16.786779+0000","last_undegraded":"2024-09-17T07:52:16.786779+0000","last_fullsized":"2024-09-17T07:52:16.786779+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:27.731564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782275+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:15.782275+0000","last_peered":"2024-09-17T07:52:15.782275+0000","last_clean":"2024-09-17T07:52:15.782275+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:15.782275+0000","last_undegraded":"2024-09-17T07:52:15.782275+0000","last_fullsized":"2024-09-17T07:52:15.782275+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":364,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.781944+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:15.781944+0000","last_peered":"2024-09-17T07:52:15.781944+0000","last_clean":"2024-09-17T07:52:15.781944+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:15.781944+0000","last_undegraded":"2024-09-17T07:52:15.781944+0000","last_fullsized":"2024-09-17T07:52:15.781944+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":101,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786702+0000","last_change":"2024-09-17T07:52:16.786930+0000","last_active":"2024-09-17T07:50:47.515749+0000","last_peered":"2024-09-17T07:46:53.908455+0000","last_clean":"2024-09-17T07:46:53.908455+0000","last_became_active":"2024-09-17T07:46:52.995980+0000","last_became_peered":"2024-09-17T07:46:52.995980+0000","last_unstale":"2024-09-17T07:52:16.786702+0000","last_undegraded":"2024-09-17T07:52:16.786702+0000","last_fullsized":"2024-09-17T07:52:16.786702+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:59.966450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":401,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786901+0000","last_change":"2024-09-17T07:52:16.787057+0000","last_active":"2024-09-17T07:52:16.786870+0000","last_peered":"2024-09-17T07:51:55.828289+0000","last_clean":"2024-09-17T07:51:55.828289+0000","last_became_active":"2024-09-17T07:50:50.514255+0000","last_became_peered":"2024-09-17T07:50:50.514255+0000","last_unstale":"2024-09-17T07:52:16.786901+0000","last_undegraded":"2024-09-17T07:52:16.786901+0000","last_fullsized":"2024-09-17T07:52:16.786901+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:19.570284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782138+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:15.782138+0000","last_peered":"2024-09-17T07:52:15.782138+0000","last_clean":"2024-09-17T07:52:15.782138+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:15.782138+0000","last_undegraded":"2024-09-17T07:52:15.782138+0000","last_fullsized":"2024-09-17T07:52:15.782138+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":218,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.782024+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:15.782024+0000","last_peered":"2024-09-17T07:52:15.782024+0000","last_clean":"2024-09-17T07:52:15.782024+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:15.782024+0000","last_undegraded":"2024-09-17T07:52:15.782024+0000","last_fullsized":"2024-09-17T07:52:15.782024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":98,"reported_epoch":183,"state":"peering","last_fresh":"2024-09-17T07:52:16.786821+0000","last_change":"2024-09-17T07:52:16.787033+0000","last_active":"2024-09-17T07:52:16.786793+0000","last_peered":"2024-09-17T07:51:55.702144+0000","last_clean":"2024-09-17T07:51:55.702144+0000","last_became_active":"2024-09-17T07:50:52.525533+0000","last_became_peered":"2024-09-17T07:50:52.525533+0000","last_unstale":"2024-09-17T07:52:16.786821+0000","last_undegraded":"2024-09-17T07:52:16.786821+0000","last_fullsized":"2024-09-17T07:52:16.786821+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:20.762102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337176,"kb_used_data":39052,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034664,"statfs":{"total":96636764160,"available":96291495936,"internally_reserved":0,"allocated":39989248,"data_stored":38708546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.505},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.46400000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.379}]}]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307832,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064008,"statfs":{"total":96636764160,"available":96321544192,"internally_reserved":0,"allocated":2207744,"data_stored":959483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.624}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.65400000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.49099999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.42899999999999999}]}]},{"osd":3,"up_from":126,"seq":541165879316,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329384,"kb_used_data":31444,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042456,"statfs":{"total":96636764160,"available":96299474944,"internally_reserved":0,"allocated":32198656,"data_stored":30915289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.59499999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.625}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.48199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320156,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051684,"statfs":{"total":96636764160,"available":96308924416,"internally_reserved":0,"allocated":14819328,"data_stored":13547671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.247},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.35399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.432}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38900000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640326,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361704,"kb_used_data":56020,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010136,"statfs":{"total":96636764160,"available":96266379264,"internally_reserved":0,"allocated":57364480,"data_stored":56076336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.253}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.311}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54500000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.47699999999999998}]}]},{"osd":0,"up_from":121,"seq":519691042837,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324892,"kb_used_data":26764,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046948,"statfs":{"total":96636764160,"available":96304074752,"internally_reserved":0,"allocated":27406336,"data_stored":26130579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.45000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.623}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:25.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:25.485 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:25.497 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-17T07:52:25.475624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981236,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249804,"statfs":{"total":579820584960,"available":577791799296,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.690470"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":115,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792024+0000","last_change":"2024-09-17T07:52:17.792024+0000","last_active":"2024-09-17T07:52:17.792024+0000","last_peered":"2024-09-17T07:52:17.792024+0000","last_clean":"2024-09-17T07:52:17.792024+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:17.792024+0000","last_undegraded":"2024-09-17T07:52:17.792024+0000","last_fullsized":"2024-09-17T07:52:17.792024+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:22.162755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":901,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.787472+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:16.787472+0000","last_peered":"2024-09-17T07:52:16.787472+0000","last_clean":"2024-09-17T07:52:16.787472+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:16.787472+0000","last_undegraded":"2024-09-17T07:52:16.787472+0000","last_fullsized":"2024-09-17T07:52:16.787472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2213,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954370+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:15.954370+0000","last_peered":"2024-09-17T07:52:15.954370+0000","last_clean":"2024-09-17T07:52:15.954370+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:15.954370+0000","last_undegraded":"2024-09-17T07:52:15.954370+0000","last_fullsized":"2024-09-17T07:52:15.954370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.798242+0000","last_change":"2024-09-17T07:52:17.798242+0000","last_active":"2024-09-17T07:52:17.798242+0000","last_peered":"2024-09-17T07:52:17.798242+0000","last_clean":"2024-09-17T07:52:17.798242+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:17.798242+0000","last_undegraded":"2024-09-17T07:52:17.798242+0000","last_fullsized":"2024-09-17T07:52:17.798242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:49.664564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786838+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:16.786838+0000","last_peered":"2024-09-17T07:52:16.786838+0000","last_clean":"2024-09-17T07:52:16.786838+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:16.786838+0000","last_undegraded":"2024-09-17T07:52:16.786838+0000","last_fullsized":"2024-09-17T07:52:16.786838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":365,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786933+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:16.786933+0000","last_peered":"2024-09-17T07:52:16.786933+0000","last_clean":"2024-09-17T07:52:16.786933+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:16.786933+0000","last_undegraded":"2024-09-17T07:52:16.786933+0000","last_fullsized":"2024-09-17T07:52:16.786933+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":109,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.797330+0000","last_change":"2024-09-17T07:52:17.797330+0000","last_active":"2024-09-17T07:52:17.797330+0000","last_peered":"2024-09-17T07:52:17.797330+0000","last_clean":"2024-09-17T07:52:17.797330+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:17.797330+0000","last_undegraded":"2024-09-17T07:52:17.797330+0000","last_fullsized":"2024-09-17T07:52:17.797330+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.131144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":409,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792833+0000","last_change":"2024-09-17T07:52:17.792833+0000","last_active":"2024-09-17T07:52:17.792833+0000","last_peered":"2024-09-17T07:52:17.792833+0000","last_clean":"2024-09-17T07:52:17.792833+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:17.792833+0000","last_undegraded":"2024-09-17T07:52:17.792833+0000","last_fullsized":"2024-09-17T07:52:17.792833+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:16.580020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786837+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:16.786837+0000","last_peered":"2024-09-17T07:52:16.786837+0000","last_clean":"2024-09-17T07:52:16.786837+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:16.786837+0000","last_undegraded":"2024-09-17T07:52:16.786837+0000","last_fullsized":"2024-09-17T07:52:16.786837+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":219,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786975+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:16.786975+0000","last_peered":"2024-09-17T07:52:16.786975+0000","last_clean":"2024-09-17T07:52:16.786975+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:16.786975+0000","last_undegraded":"2024-09-17T07:52:16.786975+0000","last_fullsized":"2024-09-17T07:52:16.786975+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":106,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792192+0000","last_change":"2024-09-17T07:52:17.792192+0000","last_active":"2024-09-17T07:52:17.792192+0000","last_peered":"2024-09-17T07:52:17.792192+0000","last_clean":"2024-09-17T07:52:17.792192+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:17.792192+0000","last_undegraded":"2024-09-17T07:52:17.792192+0000","last_fullsized":"2024-09-17T07:52:17.792192+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:26.554015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337200,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034640,"statfs":{"total":96636764160,"available":96291471360,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.436}]}]},{"osd":4,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307856,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063984,"statfs":{"total":96636764160,"available":96321519616,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.36499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.309}]}]},{"osd":3,"up_from":126,"seq":541165879317,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329388,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042452,"statfs":{"total":96636764160,"available":96299470848,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.34399999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.46700000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320160,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051680,"statfs":{"total":96636764160,"available":96308920320,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.441}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.59799999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640327,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361724,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010116,"statfs":{"total":96636764160,"available":96266358784,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.248},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.214},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.439}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003}]}]},{"osd":0,"up_from":121,"seq":519691042838,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324908,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046932,"statfs":{"total":96636764160,"available":96304058368,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.24099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:25.499 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-17T07:52:25.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-17T07:52:25.787 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T07:52:25.795 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-17T07:52:25.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:52:26.068 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:26.069 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":184,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:52:17.784472+0000","last_up_change":"2024-09-17T07:50:48.503993+0000","last_in_change":"2024-09-17T07:52:04.724383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":38,"up_from":121,"up_thru":154,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":69763},{"type":"v1","addr":"172.21.15.26:6801","nonce":69763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":69763},{"type":"v1","addr":"172.21.15.26:6803","nonce":69763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":69763},{"type":"v1","addr":"172.21.15.26:6807","nonce":69763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":69763},{"type":"v1","addr":"172.21.15.26:6805","nonce":69763}]},"public_addr":"172.21.15.26:6801/69763","cluster_addr":"172.21.15.26:6803/69763","heartbeat_back_addr":"172.21.15.26:6807/69763","heartbeat_front_addr":"172.21.15.26:6805/69763","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":49,"up_from":126,"up_thru":183,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":61678},{"type":"v1","addr":"172.21.15.94:6817","nonce":61678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":61678},{"type":"v1","addr":"172.21.15.94:6819","nonce":61678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":61678},{"type":"v1","addr":"172.21.15.94:6823","nonce":61678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":61678},{"type":"v1","addr":"172.21.15.94:6821","nonce":61678}]},"public_addr":"172.21.15.94:6817/61678","cluster_addr":"172.21.15.94:6819/61678","heartbeat_back_addr":"172.21.15.94:6823/61678","heartbeat_front_addr":"172.21.15.94:6821/61678","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":68,"up_from":96,"up_thru":171,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":57427},{"type":"v1","addr":"172.21.15.94:6801","nonce":57427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":57427},{"type":"v1","addr":"172.21.15.94:6803","nonce":57427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":57427},{"type":"v1","addr":"172.21.15.94:6807","nonce":57427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":57427},{"type":"v1","addr":"172.21.15.94:6805","nonce":57427}]},"public_addr":"172.21.15.94:6801/57427","cluster_addr":"172.21.15.94:6803/57427","heartbeat_back_addr":"172.21.15.94:6807/57427","heartbeat_front_addr":"172.21.15.94:6805/57427","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":50},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":69}],"pg_upmap":[{"pgid":"1.2","osds":[3,1]},{"pgid":"1.5","osds":[1,0]},{"pgid":"3.1","osds":[3,2]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:52:26.082 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T07:52:26.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:26.350 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-17T07:52:25.475624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981236,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249804,"statfs":{"total":579820584960,"available":577791799296,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.690470"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":115,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792024+0000","last_change":"2024-09-17T07:52:17.792024+0000","last_active":"2024-09-17T07:52:17.792024+0000","last_peered":"2024-09-17T07:52:17.792024+0000","last_clean":"2024-09-17T07:52:17.792024+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:17.792024+0000","last_undegraded":"2024-09-17T07:52:17.792024+0000","last_fullsized":"2024-09-17T07:52:17.792024+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:22.162755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":901,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.787472+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:16.787472+0000","last_peered":"2024-09-17T07:52:16.787472+0000","last_clean":"2024-09-17T07:52:16.787472+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:16.787472+0000","last_undegraded":"2024-09-17T07:52:16.787472+0000","last_fullsized":"2024-09-17T07:52:16.787472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2213,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954370+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:15.954370+0000","last_peered":"2024-09-17T07:52:15.954370+0000","last_clean":"2024-09-17T07:52:15.954370+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:15.954370+0000","last_undegraded":"2024-09-17T07:52:15.954370+0000","last_fullsized":"2024-09-17T07:52:15.954370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.798242+0000","last_change":"2024-09-17T07:52:17.798242+0000","last_active":"2024-09-17T07:52:17.798242+0000","last_peered":"2024-09-17T07:52:17.798242+0000","last_clean":"2024-09-17T07:52:17.798242+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:17.798242+0000","last_undegraded":"2024-09-17T07:52:17.798242+0000","last_fullsized":"2024-09-17T07:52:17.798242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:49.664564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786838+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:16.786838+0000","last_peered":"2024-09-17T07:52:16.786838+0000","last_clean":"2024-09-17T07:52:16.786838+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:16.786838+0000","last_undegraded":"2024-09-17T07:52:16.786838+0000","last_fullsized":"2024-09-17T07:52:16.786838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":365,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786933+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:16.786933+0000","last_peered":"2024-09-17T07:52:16.786933+0000","last_clean":"2024-09-17T07:52:16.786933+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:16.786933+0000","last_undegraded":"2024-09-17T07:52:16.786933+0000","last_fullsized":"2024-09-17T07:52:16.786933+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":109,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.797330+0000","last_change":"2024-09-17T07:52:17.797330+0000","last_active":"2024-09-17T07:52:17.797330+0000","last_peered":"2024-09-17T07:52:17.797330+0000","last_clean":"2024-09-17T07:52:17.797330+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:17.797330+0000","last_undegraded":"2024-09-17T07:52:17.797330+0000","last_fullsized":"2024-09-17T07:52:17.797330+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.131144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":409,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792833+0000","last_change":"2024-09-17T07:52:17.792833+0000","last_active":"2024-09-17T07:52:17.792833+0000","last_peered":"2024-09-17T07:52:17.792833+0000","last_clean":"2024-09-17T07:52:17.792833+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:17.792833+0000","last_undegraded":"2024-09-17T07:52:17.792833+0000","last_fullsized":"2024-09-17T07:52:17.792833+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:16.580020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786837+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:16.786837+0000","last_peered":"2024-09-17T07:52:16.786837+0000","last_clean":"2024-09-17T07:52:16.786837+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:16.786837+0000","last_undegraded":"2024-09-17T07:52:16.786837+0000","last_fullsized":"2024-09-17T07:52:16.786837+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":219,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786975+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:16.786975+0000","last_peered":"2024-09-17T07:52:16.786975+0000","last_clean":"2024-09-17T07:52:16.786975+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:16.786975+0000","last_undegraded":"2024-09-17T07:52:16.786975+0000","last_fullsized":"2024-09-17T07:52:16.786975+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":106,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792192+0000","last_change":"2024-09-17T07:52:17.792192+0000","last_active":"2024-09-17T07:52:17.792192+0000","last_peered":"2024-09-17T07:52:17.792192+0000","last_clean":"2024-09-17T07:52:17.792192+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:17.792192+0000","last_undegraded":"2024-09-17T07:52:17.792192+0000","last_fullsized":"2024-09-17T07:52:17.792192+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:26.554015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337200,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034640,"statfs":{"total":96636764160,"available":96291471360,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.436}]}]},{"osd":4,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307856,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063984,"statfs":{"total":96636764160,"available":96321519616,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.36499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.309}]}]},{"osd":3,"up_from":126,"seq":541165879317,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329388,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042452,"statfs":{"total":96636764160,"available":96299470848,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.34399999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.46700000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320160,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051680,"statfs":{"total":96636764160,"available":96308920320,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.441}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.59799999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640327,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361724,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010116,"statfs":{"total":96636764160,"available":96266358784,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.248},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.214},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.439}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003}]}]},{"osd":0,"up_from":121,"seq":519691042838,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324908,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046932,"statfs":{"total":96636764160,"available":96304058368,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.24099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:26.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:26.632 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:26.643 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-17T07:52:25.475624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981236,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249804,"statfs":{"total":579820584960,"available":577791799296,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.690470"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":115,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792024+0000","last_change":"2024-09-17T07:52:17.792024+0000","last_active":"2024-09-17T07:52:17.792024+0000","last_peered":"2024-09-17T07:52:17.792024+0000","last_clean":"2024-09-17T07:52:17.792024+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:17.792024+0000","last_undegraded":"2024-09-17T07:52:17.792024+0000","last_fullsized":"2024-09-17T07:52:17.792024+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:22.162755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":901,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.787472+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:16.787472+0000","last_peered":"2024-09-17T07:52:16.787472+0000","last_clean":"2024-09-17T07:52:16.787472+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:16.787472+0000","last_undegraded":"2024-09-17T07:52:16.787472+0000","last_fullsized":"2024-09-17T07:52:16.787472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2213,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954370+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:15.954370+0000","last_peered":"2024-09-17T07:52:15.954370+0000","last_clean":"2024-09-17T07:52:15.954370+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:15.954370+0000","last_undegraded":"2024-09-17T07:52:15.954370+0000","last_fullsized":"2024-09-17T07:52:15.954370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.798242+0000","last_change":"2024-09-17T07:52:17.798242+0000","last_active":"2024-09-17T07:52:17.798242+0000","last_peered":"2024-09-17T07:52:17.798242+0000","last_clean":"2024-09-17T07:52:17.798242+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:17.798242+0000","last_undegraded":"2024-09-17T07:52:17.798242+0000","last_fullsized":"2024-09-17T07:52:17.798242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:49.664564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786838+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:16.786838+0000","last_peered":"2024-09-17T07:52:16.786838+0000","last_clean":"2024-09-17T07:52:16.786838+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:16.786838+0000","last_undegraded":"2024-09-17T07:52:16.786838+0000","last_fullsized":"2024-09-17T07:52:16.786838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":365,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786933+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:16.786933+0000","last_peered":"2024-09-17T07:52:16.786933+0000","last_clean":"2024-09-17T07:52:16.786933+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:16.786933+0000","last_undegraded":"2024-09-17T07:52:16.786933+0000","last_fullsized":"2024-09-17T07:52:16.786933+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":109,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.797330+0000","last_change":"2024-09-17T07:52:17.797330+0000","last_active":"2024-09-17T07:52:17.797330+0000","last_peered":"2024-09-17T07:52:17.797330+0000","last_clean":"2024-09-17T07:52:17.797330+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:17.797330+0000","last_undegraded":"2024-09-17T07:52:17.797330+0000","last_fullsized":"2024-09-17T07:52:17.797330+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.131144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":409,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792833+0000","last_change":"2024-09-17T07:52:17.792833+0000","last_active":"2024-09-17T07:52:17.792833+0000","last_peered":"2024-09-17T07:52:17.792833+0000","last_clean":"2024-09-17T07:52:17.792833+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:17.792833+0000","last_undegraded":"2024-09-17T07:52:17.792833+0000","last_fullsized":"2024-09-17T07:52:17.792833+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:16.580020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786837+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:16.786837+0000","last_peered":"2024-09-17T07:52:16.786837+0000","last_clean":"2024-09-17T07:52:16.786837+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:16.786837+0000","last_undegraded":"2024-09-17T07:52:16.786837+0000","last_fullsized":"2024-09-17T07:52:16.786837+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":219,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786975+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:16.786975+0000","last_peered":"2024-09-17T07:52:16.786975+0000","last_clean":"2024-09-17T07:52:16.786975+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:16.786975+0000","last_undegraded":"2024-09-17T07:52:16.786975+0000","last_fullsized":"2024-09-17T07:52:16.786975+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":106,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792192+0000","last_change":"2024-09-17T07:52:17.792192+0000","last_active":"2024-09-17T07:52:17.792192+0000","last_peered":"2024-09-17T07:52:17.792192+0000","last_clean":"2024-09-17T07:52:17.792192+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:17.792192+0000","last_undegraded":"2024-09-17T07:52:17.792192+0000","last_fullsized":"2024-09-17T07:52:17.792192+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:26.554015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337200,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034640,"statfs":{"total":96636764160,"available":96291471360,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.436}]}]},{"osd":4,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307856,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063984,"statfs":{"total":96636764160,"available":96321519616,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.36499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.309}]}]},{"osd":3,"up_from":126,"seq":541165879317,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329388,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042452,"statfs":{"total":96636764160,"available":96299470848,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.34399999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.46700000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320160,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051680,"statfs":{"total":96636764160,"available":96308920320,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.441}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.59799999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640327,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361724,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010116,"statfs":{"total":96636764160,"available":96266358784,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.248},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.214},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.439}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003}]}]},{"osd":0,"up_from":121,"seq":519691042838,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324908,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046932,"statfs":{"total":96636764160,"available":96304058368,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.24099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:26.645 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T07:52:26.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:26.921 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:26.934 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-17T07:52:25.475624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981236,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249804,"statfs":{"total":579820584960,"available":577791799296,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-8388665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.690470"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":115,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792024+0000","last_change":"2024-09-17T07:52:17.792024+0000","last_active":"2024-09-17T07:52:17.792024+0000","last_peered":"2024-09-17T07:52:17.792024+0000","last_clean":"2024-09-17T07:52:17.792024+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:17.792024+0000","last_undegraded":"2024-09-17T07:52:17.792024+0000","last_fullsized":"2024-09-17T07:52:17.792024+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:22.162755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":901,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.787472+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:16.787472+0000","last_peered":"2024-09-17T07:52:16.787472+0000","last_clean":"2024-09-17T07:52:16.787472+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:16.787472+0000","last_undegraded":"2024-09-17T07:52:16.787472+0000","last_fullsized":"2024-09-17T07:52:16.787472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2213,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954370+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:15.954370+0000","last_peered":"2024-09-17T07:52:15.954370+0000","last_clean":"2024-09-17T07:52:15.954370+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:15.954370+0000","last_undegraded":"2024-09-17T07:52:15.954370+0000","last_fullsized":"2024-09-17T07:52:15.954370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.798242+0000","last_change":"2024-09-17T07:52:17.798242+0000","last_active":"2024-09-17T07:52:17.798242+0000","last_peered":"2024-09-17T07:52:17.798242+0000","last_clean":"2024-09-17T07:52:17.798242+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:17.798242+0000","last_undegraded":"2024-09-17T07:52:17.798242+0000","last_fullsized":"2024-09-17T07:52:17.798242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:49.664564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786838+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:16.786838+0000","last_peered":"2024-09-17T07:52:16.786838+0000","last_clean":"2024-09-17T07:52:16.786838+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:16.786838+0000","last_undegraded":"2024-09-17T07:52:16.786838+0000","last_fullsized":"2024-09-17T07:52:16.786838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":365,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786933+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:16.786933+0000","last_peered":"2024-09-17T07:52:16.786933+0000","last_clean":"2024-09-17T07:52:16.786933+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:16.786933+0000","last_undegraded":"2024-09-17T07:52:16.786933+0000","last_fullsized":"2024-09-17T07:52:16.786933+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":109,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.797330+0000","last_change":"2024-09-17T07:52:17.797330+0000","last_active":"2024-09-17T07:52:17.797330+0000","last_peered":"2024-09-17T07:52:17.797330+0000","last_clean":"2024-09-17T07:52:17.797330+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:17.797330+0000","last_undegraded":"2024-09-17T07:52:17.797330+0000","last_fullsized":"2024-09-17T07:52:17.797330+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.131144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":409,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792833+0000","last_change":"2024-09-17T07:52:17.792833+0000","last_active":"2024-09-17T07:52:17.792833+0000","last_peered":"2024-09-17T07:52:17.792833+0000","last_clean":"2024-09-17T07:52:17.792833+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:17.792833+0000","last_undegraded":"2024-09-17T07:52:17.792833+0000","last_fullsized":"2024-09-17T07:52:17.792833+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:16.580020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786837+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:16.786837+0000","last_peered":"2024-09-17T07:52:16.786837+0000","last_clean":"2024-09-17T07:52:16.786837+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:16.786837+0000","last_undegraded":"2024-09-17T07:52:16.786837+0000","last_fullsized":"2024-09-17T07:52:16.786837+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":219,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786975+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:16.786975+0000","last_peered":"2024-09-17T07:52:16.786975+0000","last_clean":"2024-09-17T07:52:16.786975+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:16.786975+0000","last_undegraded":"2024-09-17T07:52:16.786975+0000","last_fullsized":"2024-09-17T07:52:16.786975+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":106,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792192+0000","last_change":"2024-09-17T07:52:17.792192+0000","last_active":"2024-09-17T07:52:17.792192+0000","last_peered":"2024-09-17T07:52:17.792192+0000","last_clean":"2024-09-17T07:52:17.792192+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:17.792192+0000","last_undegraded":"2024-09-17T07:52:17.792192+0000","last_fullsized":"2024-09-17T07:52:17.792192+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:26.554015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337200,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034640,"statfs":{"total":96636764160,"available":96291471360,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.436}]}]},{"osd":4,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307856,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063984,"statfs":{"total":96636764160,"available":96321519616,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.36499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.309}]}]},{"osd":3,"up_from":126,"seq":541165879317,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329388,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042452,"statfs":{"total":96636764160,"available":96299470848,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.34399999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.46700000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320160,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051680,"statfs":{"total":96636764160,"available":96308920320,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.441}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51700000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.59799999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640327,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361724,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010116,"statfs":{"total":96636764160,"available":96266358784,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.248},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.214},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.439}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003}]}]},{"osd":0,"up_from":121,"seq":519691042838,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324908,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046932,"statfs":{"total":96636764160,"available":96304058368,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.24099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:26.935 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T07:52:27.205 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:27.206 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":184,"fsid":"666af130-c9e0-433a-95ce-0b7c815e9e68","created":"2024-09-17T07:46:41.695689+0000","modified":"2024-09-17T07:52:17.784472+0000","last_up_change":"2024-09-17T07:50:48.503993+0000","last_in_change":"2024-09-17T07:52:04.724383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-17T07:46:51.441250+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-17T07:46:51.470493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-17T07:47:00.696161+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"20","last_force_op_resend_prenautilus":"20","last_force_op_resend_preluminous":"20","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b6a34e0-ea77-4062-9ebc-1c2b50cb21d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":38,"up_from":121,"up_thru":154,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":69763},{"type":"v1","addr":"172.21.15.26:6801","nonce":69763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":69763},{"type":"v1","addr":"172.21.15.26:6803","nonce":69763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":69763},{"type":"v1","addr":"172.21.15.26:6807","nonce":69763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":69763},{"type":"v1","addr":"172.21.15.26:6805","nonce":69763}]},"public_addr":"172.21.15.26:6801/69763","cluster_addr":"172.21.15.26:6803/69763","heartbeat_back_addr":"172.21.15.26:6807/69763","heartbeat_front_addr":"172.21.15.26:6805/69763","state":["exists","up"]},{"osd":1,"uuid":"b26da222-1990-4e5d-a449-0ff9d561090f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":31885},{"type":"v1","addr":"172.21.15.26:6809","nonce":31885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":31885},{"type":"v1","addr":"172.21.15.26:6812","nonce":31885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":31885},{"type":"v1","addr":"172.21.15.26:6820","nonce":31885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":31885},{"type":"v1","addr":"172.21.15.26:6816","nonce":31885}]},"public_addr":"172.21.15.26:6809/31885","cluster_addr":"172.21.15.26:6812/31885","heartbeat_back_addr":"172.21.15.26:6820/31885","heartbeat_front_addr":"172.21.15.26:6816/31885","state":["exists","up"]},{"osd":2,"uuid":"e6761959-1933-4d0a-a0ff-b437b983611e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6811","nonce":31886},{"type":"v1","addr":"172.21.15.26:6813","nonce":31886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6815","nonce":31886},{"type":"v1","addr":"172.21.15.26:6817","nonce":31886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":31886},{"type":"v1","addr":"172.21.15.26:6823","nonce":31886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6819","nonce":31886},{"type":"v1","addr":"172.21.15.26:6821","nonce":31886}]},"public_addr":"172.21.15.26:6813/31886","cluster_addr":"172.21.15.26:6817/31886","heartbeat_back_addr":"172.21.15.26:6823/31886","heartbeat_front_addr":"172.21.15.26:6821/31886","state":["exists","up"]},{"osd":3,"uuid":"5166dbfd-44ec-4d31-b486-b151dfde1443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":49,"up_from":126,"up_thru":183,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":61678},{"type":"v1","addr":"172.21.15.94:6817","nonce":61678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":61678},{"type":"v1","addr":"172.21.15.94:6819","nonce":61678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":61678},{"type":"v1","addr":"172.21.15.94:6823","nonce":61678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":61678},{"type":"v1","addr":"172.21.15.94:6821","nonce":61678}]},"public_addr":"172.21.15.94:6817/61678","cluster_addr":"172.21.15.94:6819/61678","heartbeat_back_addr":"172.21.15.94:6823/61678","heartbeat_front_addr":"172.21.15.94:6821/61678","state":["exists","up"]},{"osd":4,"uuid":"f7829bfc-4efc-4bc9-8c4d-423690de31fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":31605},{"type":"v1","addr":"172.21.15.94:6809","nonce":31605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":31605},{"type":"v1","addr":"172.21.15.94:6811","nonce":31605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":31605},{"type":"v1","addr":"172.21.15.94:6815","nonce":31605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":31605},{"type":"v1","addr":"172.21.15.94:6813","nonce":31605}]},"public_addr":"172.21.15.94:6809/31605","cluster_addr":"172.21.15.94:6811/31605","heartbeat_back_addr":"172.21.15.94:6815/31605","heartbeat_front_addr":"172.21.15.94:6813/31605","state":["exists","up"]},{"osd":5,"uuid":"4e0679f8-a785-4804-93d3-79b7acabe009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":68,"up_from":96,"up_thru":171,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":57427},{"type":"v1","addr":"172.21.15.94:6801","nonce":57427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":57427},{"type":"v1","addr":"172.21.15.94:6803","nonce":57427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":57427},{"type":"v1","addr":"172.21.15.94:6807","nonce":57427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":57427},{"type":"v1","addr":"172.21.15.94:6805","nonce":57427}]},"public_addr":"172.21.15.94:6801/57427","cluster_addr":"172.21.15.94:6803/57427","heartbeat_back_addr":"172.21.15.94:6807/57427","heartbeat_front_addr":"172.21.15.94:6805/57427","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.296842+0000","dead_epoch":39},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.328736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.376227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.427493+0000","dead_epoch":50},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.355734+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:46:48.430328+0000","dead_epoch":69}],"pg_upmap":[{"pgid":"1.2","osds":[3,1]},{"pgid":"1.5","osds":[1,0]},{"pgid":"3.1","osds":[3,2]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:52:28.219 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T07:52:28.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T07:52:28.351 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:52:28.351 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T07:52:28.352 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:52:28.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T07:52:28.633 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T07:52:28.646 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T07:52:28.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T07:52:28.764 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:52:28.764 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T07:52:28.764 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:52:28.775 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T07:52:29.049 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T07:52:29.061 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T07:52:29.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T07:52:29.180 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:52:29.181 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T07:52:29.181 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:52:29.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T07:52:29.460 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T07:52:29.474 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T07:52:29.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T07:52:29.598 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:52:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T07:52:29.598 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:52:29.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T07:52:29.882 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T07:52:29.895 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T07:52:29.895 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T07:52:30.013 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:52:30.014 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T07:52:30.014 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:52:30.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T07:52:30.296 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T07:52:30.308 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T07:52:30.308 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T07:52:30.427 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-17T07:52:30.427 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T07:52:30.427 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-17T07:52:30.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T07:52:30.707 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T07:52:30.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:30.993 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:30.993 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:31.004 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-17T07:52:29.476619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1286,"ondisk_log_size":1286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981236,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249804,"statfs":{"total":579820584960,"available":577791799296,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.690022"},"pg_stats":[{"pgid":"1.0","version":"165'13","reported_seq":115,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792024+0000","last_change":"2024-09-17T07:52:17.792024+0000","last_active":"2024-09-17T07:52:17.792024+0000","last_peered":"2024-09-17T07:52:17.792024+0000","last_clean":"2024-09-17T07:52:17.792024+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:17.792024+0000","last_undegraded":"2024-09-17T07:52:17.792024+0000","last_fullsized":"2024-09-17T07:52:17.792024+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:22.162755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"165'73","reported_seq":901,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.787472+0000","last_change":"2024-09-17T07:52:05.848458+0000","last_active":"2024-09-17T07:52:16.787472+0000","last_peered":"2024-09-17T07:52:16.787472+0000","last_clean":"2024-09-17T07:52:16.787472+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:16.787472+0000","last_undegraded":"2024-09-17T07:52:16.787472+0000","last_fullsized":"2024-09-17T07:52:16.787472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:13.336696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"165'7","reported_seq":28,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954068+0000","last_change":"2024-09-17T07:52:05.791644+0000","last_active":"2024-09-17T07:52:15.954068+0000","last_peered":"2024-09-17T07:52:15.954068+0000","last_clean":"2024-09-17T07:52:15.954068+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:15.954068+0000","last_undegraded":"2024-09-17T07:52:15.954068+0000","last_fullsized":"2024-09-17T07:52:15.954068+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:36.862427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"165'487","reported_seq":2213,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954370+0000","last_change":"2024-09-17T07:51:55.701233+0000","last_active":"2024-09-17T07:52:15.954370+0000","last_peered":"2024-09-17T07:52:15.954370+0000","last_clean":"2024-09-17T07:52:15.954370+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:15.954370+0000","last_undegraded":"2024-09-17T07:52:15.954370+0000","last_fullsized":"2024-09-17T07:52:15.954370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:26:31.413985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"165'7","reported_seq":13,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.798242+0000","last_change":"2024-09-17T07:52:17.798242+0000","last_active":"2024-09-17T07:52:17.798242+0000","last_peered":"2024-09-17T07:52:17.798242+0000","last_clean":"2024-09-17T07:52:17.798242+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:17.798242+0000","last_undegraded":"2024-09-17T07:52:17.798242+0000","last_fullsized":"2024-09-17T07:52:17.798242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:49.664564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"165'460","reported_seq":2308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786838+0000","last_change":"2024-09-17T07:51:55.705816+0000","last_active":"2024-09-17T07:52:16.786838+0000","last_peered":"2024-09-17T07:52:16.786838+0000","last_clean":"2024-09-17T07:52:16.786838+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:16.786838+0000","last_undegraded":"2024-09-17T07:52:16.786838+0000","last_fullsized":"2024-09-17T07:52:16.786838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:20.056981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":365,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786933+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:16.786933+0000","last_peered":"2024-09-17T07:52:16.786933+0000","last_clean":"2024-09-17T07:52:16.786933+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:16.786933+0000","last_undegraded":"2024-09-17T07:52:16.786933+0000","last_fullsized":"2024-09-17T07:52:16.786933+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:42.174828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'72","reported_seq":109,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.797330+0000","last_change":"2024-09-17T07:52:17.797330+0000","last_active":"2024-09-17T07:52:17.797330+0000","last_peered":"2024-09-17T07:52:17.797330+0000","last_clean":"2024-09-17T07:52:17.797330+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:17.797330+0000","last_undegraded":"2024-09-17T07:52:17.797330+0000","last_fullsized":"2024-09-17T07:52:17.797330+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.131144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"165'123","reported_seq":409,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792833+0000","last_change":"2024-09-17T07:52:17.792833+0000","last_active":"2024-09-17T07:52:17.792833+0000","last_peered":"2024-09-17T07:52:17.792833+0000","last_clean":"2024-09-17T07:52:17.792833+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:17.792833+0000","last_undegraded":"2024-09-17T07:52:17.792833+0000","last_fullsized":"2024-09-17T07:52:17.792833+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:47:01.286796+0000","last_clean_scrub_stamp":"2024-09-17T07:47:01.286796+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:16.580020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"165'10","reported_seq":308,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786837+0000","last_change":"2024-09-17T07:51:55.704939+0000","last_active":"2024-09-17T07:52:16.786837+0000","last_peered":"2024-09-17T07:52:16.786837+0000","last_clean":"2024-09-17T07:52:16.786837+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:16.786837+0000","last_undegraded":"2024-09-17T07:52:16.786837+0000","last_fullsized":"2024-09-17T07:52:16.786837+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:26.785917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"165'9","reported_seq":219,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:52:16.786975+0000","last_change":"2024-09-17T07:51:55.705087+0000","last_active":"2024-09-17T07:52:16.786975+0000","last_peered":"2024-09-17T07:52:16.786975+0000","last_clean":"2024-09-17T07:52:16.786975+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:16.786975+0000","last_undegraded":"2024-09-17T07:52:16.786975+0000","last_fullsized":"2024-09-17T07:52:16.786975+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:10:01.133938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"165'4","reported_seq":106,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.792192+0000","last_change":"2024-09-17T07:52:17.792192+0000","last_active":"2024-09-17T07:52:17.792192+0000","last_peered":"2024-09-17T07:52:17.792192+0000","last_clean":"2024-09-17T07:52:17.792192+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:17.792192+0000","last_undegraded":"2024-09-17T07:52:17.792192+0000","last_fullsized":"2024-09-17T07:52:17.792192+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:26.554015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"45'5","reported_seq":30,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:52:15.954114+0000","last_change":"2024-09-17T07:52:05.786854+0000","last_active":"2024-09-17T07:52:15.954114+0000","last_peered":"2024-09-17T07:52:15.954114+0000","last_clean":"2024-09-17T07:52:15.954114+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:15.954114+0000","last_undegraded":"2024-09-17T07:52:15.954114+0000","last_fullsized":"2024-09-17T07:52:15.954114+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:04:54.370311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860449,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337200,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034640,"statfs":{"total":96636764160,"available":96291471360,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.436}]}]},{"osd":4,"up_from":11,"seq":47244640327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307856,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063984,"statfs":{"total":96636764160,"available":96321519616,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.442}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38300000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.42499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.36099999999999999}]}]},{"osd":3,"up_from":126,"seq":541165879318,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329388,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042452,"statfs":{"total":96636764160,"available":96299470848,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.379},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.436}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41099999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.47299999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640327,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320160,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051680,"statfs":{"total":96636764160,"available":96308920320,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.309},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.32900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.28599999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.503},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45000000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.51700000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640328,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361724,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010116,"statfs":{"total":96636764160,"available":96266358784,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.47399999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.623}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.68200000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.75}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.66800000000000004}]}]},{"osd":0,"up_from":121,"seq":519691042839,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324908,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046932,"statfs":{"total":96636764160,"available":96304058368,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"last":0.248}]},{"osd":2,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"last":0.312}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.58699999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.54900000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:31.009 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.009 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-17T07:47:01.286796+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=47, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.009 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.009 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-17T07:47:01.286796+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=47, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.009 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-17T07:47:01.286796+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=47, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-17T07:47:01.286796+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=47, tm_sec=1, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.010 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:31.011 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T07:52:51.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:52:51.299 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:52:51.300 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:52:51.311 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-17T07:52:49.480912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":226,"num_omap_keys":27,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1372,"ondisk_log_size":1372,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981200,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249840,"statfs":{"total":579820584960,"available":577791836160,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":226,"num_omap_keys":27,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002683"},"pg_stats":[{"pgid":"1.0","version":"184'21","reported_seq":134,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:36.805870+0000","last_change":"2024-09-17T07:52:36.805845+0000","last_active":"2024-09-17T07:52:36.805870+0000","last_peered":"2024-09-17T07:52:36.805870+0000","last_clean":"2024-09-17T07:52:36.805870+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:36.805870+0000","last_undegraded":"2024-09-17T07:52:36.805870+0000","last_fullsized":"2024-09-17T07:52:36.805870+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'21","last_scrub_stamp":"2024-09-17T07:52:36.805759+0000","last_deep_scrub":"184'21","last_deep_scrub_stamp":"2024-09-17T07:52:36.805759+0000","last_clean_scrub_stamp":"2024-09-17T07:52:36.805759+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:30:25.631136+0000","scrub_duration":0.014119827999999999,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"184'84","reported_seq":923,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:29.595166+0000","last_change":"2024-09-17T07:52:29.595126+0000","last_active":"2024-09-17T07:52:29.595166+0000","last_peered":"2024-09-17T07:52:29.595166+0000","last_clean":"2024-09-17T07:52:29.595166+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:29.595166+0000","last_undegraded":"2024-09-17T07:52:29.595166+0000","last_fullsized":"2024-09-17T07:52:29.595166+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'84","last_scrub_stamp":"2024-09-17T07:52:29.594991+0000","last_deep_scrub":"184'84","last_deep_scrub_stamp":"2024-09-17T07:52:29.594991+0000","last_clean_scrub_stamp":"2024-09-17T07:52:29.594991+0000","objects_scrubbed":11,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:51.057723+0000","scrub_duration":0.0084941870000000003,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"184'9","reported_seq":41,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:31.213771+0000","last_change":"2024-09-17T07:52:31.213733+0000","last_active":"2024-09-17T07:52:31.213771+0000","last_peered":"2024-09-17T07:52:31.213771+0000","last_clean":"2024-09-17T07:52:31.213771+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:31.213771+0000","last_undegraded":"2024-09-17T07:52:31.213771+0000","last_fullsized":"2024-09-17T07:52:31.213771+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'9","last_scrub_stamp":"2024-09-17T07:52:31.213591+0000","last_deep_scrub":"184'9","last_deep_scrub_stamp":"2024-09-17T07:52:31.213591+0000","last_clean_scrub_stamp":"2024-09-17T07:52:31.213591+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:18.759900+0000","scrub_duration":0.0038789129999999999,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"184'507","reported_seq":2244,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:34.439495+0000","last_change":"2024-09-17T07:52:34.439450+0000","last_active":"2024-09-17T07:52:34.439495+0000","last_peered":"2024-09-17T07:52:34.439495+0000","last_clean":"2024-09-17T07:52:34.439495+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:34.439495+0000","last_undegraded":"2024-09-17T07:52:34.439495+0000","last_fullsized":"2024-09-17T07:52:34.439495+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'507","last_scrub_stamp":"2024-09-17T07:52:34.439350+0000","last_deep_scrub":"184'507","last_deep_scrub_stamp":"2024-09-17T07:52:34.439350+0000","last_clean_scrub_stamp":"2024-09-17T07:52:34.439350+0000","objects_scrubbed":20,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:59.248743+0000","scrub_duration":0.012439571999999999,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"184'11","reported_seq":28,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:31.851456+0000","last_change":"2024-09-17T07:52:31.851395+0000","last_active":"2024-09-17T07:52:31.851456+0000","last_peered":"2024-09-17T07:52:31.851456+0000","last_clean":"2024-09-17T07:52:31.851456+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:31.851456+0000","last_undegraded":"2024-09-17T07:52:31.851456+0000","last_fullsized":"2024-09-17T07:52:31.851456+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'11","last_scrub_stamp":"2024-09-17T07:52:31.851291+0000","last_deep_scrub":"184'11","last_deep_scrub_stamp":"2024-09-17T07:52:31.851291+0000","last_clean_scrub_stamp":"2024-09-17T07:52:31.851291+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:14.333344+0000","scrub_duration":0.011960558,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"184'476","reported_seq":2335,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:32.543088+0000","last_change":"2024-09-17T07:52:32.543060+0000","last_active":"2024-09-17T07:52:32.543088+0000","last_peered":"2024-09-17T07:52:32.543088+0000","last_clean":"2024-09-17T07:52:32.543088+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:32.543088+0000","last_undegraded":"2024-09-17T07:52:32.543088+0000","last_fullsized":"2024-09-17T07:52:32.543088+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'476","last_scrub_stamp":"2024-09-17T07:52:32.542961+0000","last_deep_scrub":"184'476","last_deep_scrub_stamp":"2024-09-17T07:52:32.542961+0000","last_clean_scrub_stamp":"2024-09-17T07:52:32.542961+0000","objects_scrubbed":16,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:15.912778+0000","scrub_duration":0.012423211999999999,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"165'16","reported_seq":366,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:17.791086+0000","last_change":"2024-09-17T07:52:05.836673+0000","last_active":"2024-09-17T07:52:17.791086+0000","last_peered":"2024-09-17T07:52:17.791086+0000","last_clean":"2024-09-17T07:52:17.791086+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:17.791086+0000","last_undegraded":"2024-09-17T07:52:17.791086+0000","last_fullsized":"2024-09-17T07:52:17.791086+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:46:51.988421+0000","last_clean_scrub_stamp":"2024-09-17T07:46:51.988421+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"184'74","reported_seq":122,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:30.842522+0000","last_change":"2024-09-17T07:52:30.842497+0000","last_active":"2024-09-17T07:52:30.842522+0000","last_peered":"2024-09-17T07:52:30.842522+0000","last_clean":"2024-09-17T07:52:30.842522+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:30.842522+0000","last_undegraded":"2024-09-17T07:52:30.842522+0000","last_fullsized":"2024-09-17T07:52:30.842522+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'74","last_scrub_stamp":"2024-09-17T07:52:30.842403+0000","last_deep_scrub":"184'74","last_deep_scrub_stamp":"2024-09-17T07:52:30.842403+0000","last_clean_scrub_stamp":"2024-09-17T07:52:30.842403+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:39.930183+0000","scrub_duration":0.0037516899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"184'130","reported_seq":427,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:40.844133+0000","last_change":"2024-09-17T07:52:40.844098+0000","last_active":"2024-09-17T07:52:40.844133+0000","last_peered":"2024-09-17T07:52:40.844133+0000","last_clean":"2024-09-17T07:52:40.844133+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:40.844133+0000","last_undegraded":"2024-09-17T07:52:40.844133+0000","last_fullsized":"2024-09-17T07:52:40.844133+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-09-17T07:52:40.843977+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-09-17T07:52:40.843977+0000","last_clean_scrub_stamp":"2024-09-17T07:52:40.843977+0000","objects_scrubbed":7,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:52.288599+0000","scrub_duration":0.0061816909999999996,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"184'16","reported_seq":325,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:38.495533+0000","last_change":"2024-09-17T07:52:38.495504+0000","last_active":"2024-09-17T07:52:38.495533+0000","last_peered":"2024-09-17T07:52:38.495533+0000","last_clean":"2024-09-17T07:52:38.495533+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:38.495533+0000","last_undegraded":"2024-09-17T07:52:38.495533+0000","last_fullsized":"2024-09-17T07:52:38.495533+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'16","last_scrub_stamp":"2024-09-17T07:52:38.495418+0000","last_deep_scrub":"184'16","last_deep_scrub_stamp":"2024-09-17T07:52:38.495418+0000","last_clean_scrub_stamp":"2024-09-17T07:52:38.495418+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:10.450826+0000","scrub_duration":0.01461054,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":77,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"184'12","reported_seq":233,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:42.350865+0000","last_change":"2024-09-17T07:52:42.350836+0000","last_active":"2024-09-17T07:52:42.350865+0000","last_peered":"2024-09-17T07:52:42.350865+0000","last_clean":"2024-09-17T07:52:42.350865+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:42.350865+0000","last_undegraded":"2024-09-17T07:52:42.350865+0000","last_fullsized":"2024-09-17T07:52:42.350865+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'12","last_scrub_stamp":"2024-09-17T07:52:42.350730+0000","last_deep_scrub":"184'12","last_deep_scrub_stamp":"2024-09-17T07:52:42.350730+0000","last_clean_scrub_stamp":"2024-09-17T07:52:42.350730+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:18.469209+0000","scrub_duration":0.018907486000000001,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"184'6","reported_seq":119,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:38.784442+0000","last_change":"2024-09-17T07:52:38.784418+0000","last_active":"2024-09-17T07:52:38.784442+0000","last_peered":"2024-09-17T07:52:38.784442+0000","last_clean":"2024-09-17T07:52:38.784442+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:38.784442+0000","last_undegraded":"2024-09-17T07:52:38.784442+0000","last_fullsized":"2024-09-17T07:52:38.784442+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'6","last_scrub_stamp":"2024-09-17T07:52:38.784329+0000","last_deep_scrub":"184'6","last_deep_scrub_stamp":"2024-09-17T07:52:38.784329+0000","last_clean_scrub_stamp":"2024-09-17T07:52:38.784329+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:09:08.536909+0000","scrub_duration":0.0081572710000000007,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"184'10","reported_seq":46,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:34.278717+0000","last_change":"2024-09-17T07:52:34.278669+0000","last_active":"2024-09-17T07:52:34.278717+0000","last_peered":"2024-09-17T07:52:34.278717+0000","last_clean":"2024-09-17T07:52:34.278717+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:34.278717+0000","last_undegraded":"2024-09-17T07:52:34.278717+0000","last_fullsized":"2024-09-17T07:52:34.278717+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10","last_scrub_stamp":"2024-09-17T07:52:34.278574+0000","last_deep_scrub":"184'10","last_deep_scrub_stamp":"2024-09-17T07:52:34.278574+0000","last_clean_scrub_stamp":"2024-09-17T07:52:34.278574+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:59:57.168813+0000","scrub_duration":0.016642245,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1197,"ondisk_log_size":1197,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":226,"num_omap_keys":27,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337196,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034644,"statfs":{"total":96636764160,"available":96291475456,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.51200000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61599999999999999,"5min":0.624,"15min":0.624},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.375},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.314},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.34000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307852,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063988,"statfs":{"total":96636764160,"available":96321523712,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.46700000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.55200000000000005}]},{"osd":3,"last update":"Tue Sep 17 07:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.40000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31}]}]},{"osd":3,"up_from":126,"seq":541165879322,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329384,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042456,"statfs":{"total":96636764160,"available":96299474944,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.61499999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.63600000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.49099999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320156,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051684,"statfs":{"total":96636764160,"available":96308924416,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.372},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.436,"5min":0.436,"15min":0.436},"last":0.31900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.28499999999999998,"15min":0.317},"min":{"1min":0.22800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49299999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.28599999999999998,"15min":0.30599999999999999},"min":{"1min":0.23499999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.40600000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.61099999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60499999999999998,"5min":0.628,"15min":0.628},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.62,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.626}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.67600000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640332,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361716,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010124,"statfs":{"total":96636764160,"available":96266366976,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.30199999999999999,"15min":0.33200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48099999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33100000000000002,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.222,"15min":0.222},"max":{"1min":0.49199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.443}]},{"osd":3,"last update":"Tue Sep 17 07:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.503},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.61699999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44600000000000001,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.58199999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.58899999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.52700000000000002}]}]},{"osd":0,"up_from":121,"seq":519691042843,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324896,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046944,"statfs":{"total":96636764160,"available":96304070656,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.30099999999999999,"15min":0.29099999999999998},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54600000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.29699999999999999,"15min":0.28399999999999997},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.314,"15min":0.29999999999999999},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.498},{"interface":"front","average":{"1min":0.373,"5min":0.31,"15min":0.29999999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.64500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.443,"15min":0.42599999999999999},"min":{"1min":0.38800000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.46000000000000002,"15min":0.44500000000000001},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.53400000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.45800000000000002,"15min":0.442},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.45300000000000001,"15min":0.437},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.58699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:52:51.314 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-17T07:46:51.988421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=46, tm_sec=51, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=7, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T07:52:51.314 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T07:53:11.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T07:53:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-17T07:53:11.600 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-17T07:53:11.611 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-17T07:53:11.485767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80283279,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":8637,"num_read_kb":198186,"num_write":1126,"num_write_kb":274818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":402721295,"num_keys_recovered":37,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":226,"num_omap_keys":27,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1382,"ondisk_log_size":1382,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1981200,"kb_used_data":170016,"kb_used_omap":21,"kb_used_meta":1811114,"kb_avail":564249840,"statfs":{"total":579820584960,"available":577791836160,"internally_reserved":0,"allocated":174096384,"data_stored":166383282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22231,"internal_metadata":1854581033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002849"},"pg_stats":[{"pgid":"1.0","version":"184'21","reported_seq":134,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:36.805870+0000","last_change":"2024-09-17T07:52:36.805845+0000","last_active":"2024-09-17T07:52:36.805870+0000","last_peered":"2024-09-17T07:52:36.805870+0000","last_clean":"2024-09-17T07:52:36.805870+0000","last_became_active":"2024-09-17T07:52:17.791777+0000","last_became_peered":"2024-09-17T07:52:17.791777+0000","last_unstale":"2024-09-17T07:52:36.805870+0000","last_undegraded":"2024-09-17T07:52:36.805870+0000","last_fullsized":"2024-09-17T07:52:36.805870+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'21","last_scrub_stamp":"2024-09-17T07:52:36.805759+0000","last_deep_scrub":"184'21","last_deep_scrub_stamp":"2024-09-17T07:52:36.805759+0000","last_clean_scrub_stamp":"2024-09-17T07:52:36.805759+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:30:25.631136+0000","scrub_duration":0.014119827999999999,"objects_trimmed":0,"snaptrim_duration":0.00039047300000000001,"stat_sum":{"num_bytes":8388762,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":23,"num_write":11,"num_write_kb":16389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"184'84","reported_seq":923,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:29.595166+0000","last_change":"2024-09-17T07:52:29.595126+0000","last_active":"2024-09-17T07:52:29.595166+0000","last_peered":"2024-09-17T07:52:29.595166+0000","last_clean":"2024-09-17T07:52:29.595166+0000","last_became_active":"2024-09-17T07:52:05.739364+0000","last_became_peered":"2024-09-17T07:52:05.739364+0000","last_unstale":"2024-09-17T07:52:29.595166+0000","last_undegraded":"2024-09-17T07:52:29.595166+0000","last_fullsized":"2024-09-17T07:52:29.595166+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'84","last_scrub_stamp":"2024-09-17T07:52:29.594991+0000","last_deep_scrub":"184'84","last_deep_scrub_stamp":"2024-09-17T07:52:29.594991+0000","last_clean_scrub_stamp":"2024-09-17T07:52:29.594991+0000","objects_scrubbed":11,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:51.057723+0000","scrub_duration":0.0084941870000000003,"objects_trimmed":0,"snaptrim_duration":0.001254982,"stat_sum":{"num_bytes":116,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":9,"num_read":1411,"num_read_kb":35646,"num_write":48,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":37815798,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":97,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"184'9","reported_seq":41,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:31.213771+0000","last_change":"2024-09-17T07:52:31.213733+0000","last_active":"2024-09-17T07:52:31.213771+0000","last_peered":"2024-09-17T07:52:31.213771+0000","last_clean":"2024-09-17T07:52:31.213771+0000","last_became_active":"2024-09-17T07:52:05.739385+0000","last_became_peered":"2024-09-17T07:52:05.739385+0000","last_unstale":"2024-09-17T07:52:31.213771+0000","last_undegraded":"2024-09-17T07:52:31.213771+0000","last_fullsized":"2024-09-17T07:52:31.213771+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'9","last_scrub_stamp":"2024-09-17T07:52:31.213591+0000","last_deep_scrub":"184'9","last_deep_scrub_stamp":"2024-09-17T07:52:31.213591+0000","last_clean_scrub_stamp":"2024-09-17T07:52:31.213591+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:12:18.759900+0000","scrub_duration":0.0038789129999999999,"objects_trimmed":0,"snaptrim_duration":0.00055383500000000003,"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"184'507","reported_seq":2244,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:34.439495+0000","last_change":"2024-09-17T07:52:34.439450+0000","last_active":"2024-09-17T07:52:34.439495+0000","last_peered":"2024-09-17T07:52:34.439495+0000","last_clean":"2024-09-17T07:52:34.439495+0000","last_became_active":"2024-09-17T07:50:47.508723+0000","last_became_peered":"2024-09-17T07:50:47.508723+0000","last_unstale":"2024-09-17T07:52:34.439495+0000","last_undegraded":"2024-09-17T07:52:34.439495+0000","last_fullsized":"2024-09-17T07:52:34.439495+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'507","last_scrub_stamp":"2024-09-17T07:52:34.439350+0000","last_deep_scrub":"184'507","last_deep_scrub_stamp":"2024-09-17T07:52:34.439350+0000","last_clean_scrub_stamp":"2024-09-17T07:52:34.439350+0000","objects_scrubbed":20,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:59.248743+0000","scrub_duration":0.012439571999999999,"objects_trimmed":0,"snaptrim_duration":0.000204911,"stat_sum":{"num_bytes":113,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":18,"num_read":2354,"num_read_kb":88989,"num_write":534,"num_write_kb":66624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":41943261,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":113,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"184'11","reported_seq":28,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:31.851456+0000","last_change":"2024-09-17T07:52:31.851395+0000","last_active":"2024-09-17T07:52:31.851456+0000","last_peered":"2024-09-17T07:52:31.851456+0000","last_clean":"2024-09-17T07:52:31.851456+0000","last_became_active":"2024-09-17T07:52:17.797994+0000","last_became_peered":"2024-09-17T07:52:17.797994+0000","last_unstale":"2024-09-17T07:52:31.851456+0000","last_undegraded":"2024-09-17T07:52:31.851456+0000","last_fullsized":"2024-09-17T07:52:31.851456+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'11","last_scrub_stamp":"2024-09-17T07:52:31.851291+0000","last_deep_scrub":"184'11","last_deep_scrub_stamp":"2024-09-17T07:52:31.851291+0000","last_clean_scrub_stamp":"2024-09-17T07:52:31.851291+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:14.333344+0000","scrub_duration":0.011960558,"objects_trimmed":0,"snaptrim_duration":0.000256063,"stat_sum":{"num_bytes":8388665,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":7,"num_write_kb":12291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":37748774,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"184'476","reported_seq":2335,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:32.543088+0000","last_change":"2024-09-17T07:52:32.543060+0000","last_active":"2024-09-17T07:52:32.543088+0000","last_peered":"2024-09-17T07:52:32.543088+0000","last_clean":"2024-09-17T07:52:32.543088+0000","last_became_active":"2024-09-17T07:50:17.505063+0000","last_became_peered":"2024-09-17T07:50:17.505063+0000","last_unstale":"2024-09-17T07:52:32.543088+0000","last_undegraded":"2024-09-17T07:52:32.543088+0000","last_fullsized":"2024-09-17T07:52:32.543088+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'476","last_scrub_stamp":"2024-09-17T07:52:32.542961+0000","last_deep_scrub":"184'476","last_deep_scrub_stamp":"2024-09-17T07:52:32.542961+0000","last_clean_scrub_stamp":"2024-09-17T07:52:32.542961+0000","objects_scrubbed":16,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:15.912778+0000","scrub_duration":0.012423211999999999,"objects_trimmed":0,"snaptrim_duration":0.0011548210000000001,"stat_sum":{"num_bytes":201,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":4468,"num_read_kb":48742,"num_write":258,"num_write_kb":33881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":201,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"184'26","reported_seq":386,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:47.213524+0000","last_change":"2024-09-17T07:52:47.213498+0000","last_active":"2024-09-17T07:52:47.213524+0000","last_peered":"2024-09-17T07:52:47.213524+0000","last_clean":"2024-09-17T07:52:47.213524+0000","last_became_active":"2024-09-17T07:52:05.739229+0000","last_became_peered":"2024-09-17T07:52:05.739229+0000","last_unstale":"2024-09-17T07:52:47.213524+0000","last_undegraded":"2024-09-17T07:52:47.213524+0000","last_fullsized":"2024-09-17T07:52:47.213524+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'26","last_scrub_stamp":"2024-09-17T07:52:47.213399+0000","last_deep_scrub":"184'26","last_deep_scrub_stamp":"2024-09-17T07:52:47.213399+0000","last_clean_scrub_stamp":"2024-09-17T07:52:47.213399+0000","objects_scrubbed":10,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:46.861245+0000","scrub_duration":0.031703652999999998,"objects_trimmed":0,"snaptrim_duration":0.00031824099999999998,"stat_sum":{"num_bytes":25166018,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":15,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":100663566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"184'74","reported_seq":122,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:30.842522+0000","last_change":"2024-09-17T07:52:30.842497+0000","last_active":"2024-09-17T07:52:30.842522+0000","last_peered":"2024-09-17T07:52:30.842522+0000","last_clean":"2024-09-17T07:52:30.842522+0000","last_became_active":"2024-09-17T07:52:17.797080+0000","last_became_peered":"2024-09-17T07:52:17.797080+0000","last_unstale":"2024-09-17T07:52:30.842522+0000","last_undegraded":"2024-09-17T07:52:30.842522+0000","last_fullsized":"2024-09-17T07:52:30.842522+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'74","last_scrub_stamp":"2024-09-17T07:52:30.842403+0000","last_deep_scrub":"184'74","last_deep_scrub_stamp":"2024-09-17T07:52:30.842403+0000","last_clean_scrub_stamp":"2024-09-17T07:52:30.842403+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:39.930183+0000","scrub_duration":0.0037516899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"184'130","reported_seq":427,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:40.844133+0000","last_change":"2024-09-17T07:52:40.844098+0000","last_active":"2024-09-17T07:52:40.844133+0000","last_peered":"2024-09-17T07:52:40.844133+0000","last_clean":"2024-09-17T07:52:40.844133+0000","last_became_active":"2024-09-17T07:52:17.792424+0000","last_became_peered":"2024-09-17T07:52:17.792424+0000","last_unstale":"2024-09-17T07:52:40.844133+0000","last_undegraded":"2024-09-17T07:52:40.844133+0000","last_fullsized":"2024-09-17T07:52:40.844133+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-09-17T07:52:40.843977+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-09-17T07:52:40.843977+0000","last_clean_scrub_stamp":"2024-09-17T07:52:40.843977+0000","objects_scrubbed":7,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:52.288599+0000","scrub_duration":0.0061816909999999996,"objects_trimmed":0,"snaptrim_duration":0.00052929799999999999,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":6,"num_read":255,"num_read_kb":24703,"num_write":114,"num_write_kb":25135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":25166030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":92,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"184'16","reported_seq":325,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:38.495533+0000","last_change":"2024-09-17T07:52:38.495504+0000","last_active":"2024-09-17T07:52:38.495533+0000","last_peered":"2024-09-17T07:52:38.495533+0000","last_clean":"2024-09-17T07:52:38.495533+0000","last_became_active":"2024-09-17T07:50:52.527936+0000","last_became_peered":"2024-09-17T07:50:52.527936+0000","last_unstale":"2024-09-17T07:52:38.495533+0000","last_undegraded":"2024-09-17T07:52:38.495533+0000","last_fullsized":"2024-09-17T07:52:38.495533+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'16","last_scrub_stamp":"2024-09-17T07:52:38.495418+0000","last_deep_scrub":"184'16","last_deep_scrub_stamp":"2024-09-17T07:52:38.495418+0000","last_clean_scrub_stamp":"2024-09-17T07:52:38.495418+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:10.450826+0000","scrub_duration":0.01461054,"objects_trimmed":0,"snaptrim_duration":0.000368737,"stat_sum":{"num_bytes":8388686,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":16,"num_write_kb":16394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":9,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":77,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"184'12","reported_seq":233,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:42.350865+0000","last_change":"2024-09-17T07:52:42.350836+0000","last_active":"2024-09-17T07:52:42.350865+0000","last_peered":"2024-09-17T07:52:42.350865+0000","last_clean":"2024-09-17T07:52:42.350865+0000","last_became_active":"2024-09-17T07:50:45.802092+0000","last_became_peered":"2024-09-17T07:50:45.802092+0000","last_unstale":"2024-09-17T07:52:42.350865+0000","last_undegraded":"2024-09-17T07:52:42.350865+0000","last_fullsized":"2024-09-17T07:52:42.350865+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'12","last_scrub_stamp":"2024-09-17T07:52:42.350730+0000","last_deep_scrub":"184'12","last_deep_scrub_stamp":"2024-09-17T07:52:42.350730+0000","last_clean_scrub_stamp":"2024-09-17T07:52:42.350730+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:18.469209+0000","scrub_duration":0.018907486000000001,"objects_trimmed":0,"snaptrim_duration":0.00044047200000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"184'6","reported_seq":119,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:38.784442+0000","last_change":"2024-09-17T07:52:38.784418+0000","last_active":"2024-09-17T07:52:38.784442+0000","last_peered":"2024-09-17T07:52:38.784442+0000","last_clean":"2024-09-17T07:52:38.784442+0000","last_became_active":"2024-09-17T07:52:17.791865+0000","last_became_peered":"2024-09-17T07:52:17.791865+0000","last_unstale":"2024-09-17T07:52:38.784442+0000","last_undegraded":"2024-09-17T07:52:38.784442+0000","last_fullsized":"2024-09-17T07:52:38.784442+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'6","last_scrub_stamp":"2024-09-17T07:52:38.784329+0000","last_deep_scrub":"184'6","last_deep_scrub_stamp":"2024-09-17T07:52:38.784329+0000","last_clean_scrub_stamp":"2024-09-17T07:52:38.784329+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:09:08.536909+0000","scrub_duration":0.0081572710000000007,"objects_trimmed":0,"snaptrim_duration":0.000777715,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"184'10","reported_seq":46,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-09-17T07:52:34.278717+0000","last_change":"2024-09-17T07:52:34.278669+0000","last_active":"2024-09-17T07:52:34.278717+0000","last_peered":"2024-09-17T07:52:34.278717+0000","last_clean":"2024-09-17T07:52:34.278717+0000","last_became_active":"2024-09-17T07:52:05.738124+0000","last_became_peered":"2024-09-17T07:52:05.738124+0000","last_unstale":"2024-09-17T07:52:34.278717+0000","last_undegraded":"2024-09-17T07:52:34.278717+0000","last_fullsized":"2024-09-17T07:52:34.278717+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10","last_scrub_stamp":"2024-09-17T07:52:34.278574+0000","last_deep_scrub":"184'10","last_deep_scrub_stamp":"2024-09-17T07:52:34.278574+0000","last_clean_scrub_stamp":"2024-09-17T07:52:34.278574+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:59:57.168813+0000","scrub_duration":0.016642245,"objects_trimmed":0,"snaptrim_duration":0.00038640000000000001,"stat_sum":{"num_bytes":12582966,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":37748898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":522,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":45,"num_read":8488,"num_read_kb":198080,"num_write":954,"num_write_kb":142092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":104925089,"num_keys_recovered":9,"num_objects_omap":8,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":503,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1197,"ondisk_log_size":1197,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":79692389,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":83,"num_read_kb":51,"num_write":71,"num_write_kb":131098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":297796206,"num_keys_recovered":28,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":226,"num_omap_keys":27,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167886848,"data_stored":167773443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":96,"seq":412316860457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337196,"kb_used_data":39076,"kb_used_omap":6,"kb_used_meta":298105,"kb_avail":94034644,"statfs":{"total":96636764160,"available":96291475456,"internally_reserved":0,"allocated":40013824,"data_stored":38718630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6282,"internal_metadata":305260406},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.442,"15min":0.42999999999999999},"min":{"1min":0.33300000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.436},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.36299999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.44500000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.42599999999999999,"15min":0.40999999999999998},"min":{"1min":0.39800000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.434,"15min":0.42599999999999999},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63300000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.378}]},{"osd":2,"last update":"Tue Sep 17 07:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.441,"15min":0.433},"min":{"1min":0.39000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.432,"15min":0.41999999999999998},"min":{"1min":0.36799999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36799999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.25800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.35299999999999998,"15min":0.34599999999999997},"min":{"1min":0.29399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.30499999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.34100000000000003,"15min":0.32500000000000001},"min":{"1min":0.31,"5min":0.217,"15min":0.217},"max":{"1min":0.50600000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34000000000000002,"15min":0.32100000000000001},"min":{"1min":0.28799999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28799999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307852,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063988,"statfs":{"total":96636764160,"available":96321523712,"internally_reserved":0,"allocated":2232320,"data_stored":969567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.39300000000000002},"min":{"1min":0.318,"5min":0.251,"15min":0.251},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.41899999999999998},"min":{"1min":0.377,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.50800000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.44600000000000001},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.432},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60099999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44700000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.35999999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.41399999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.371},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29599999999999999},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.35099999999999998}]}]},{"osd":3,"up_from":126,"seq":541165879326,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329384,"kb_used_data":31456,"kb_used_omap":6,"kb_used_meta":297913,"kb_avail":94042456,"statfs":{"total":96636764160,"available":96299474944,"internally_reserved":0,"allocated":32210944,"data_stored":30920331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6235,"internal_metadata":305063845},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42299999999999999,"15min":0.41499999999999998},"min":{"1min":0.34699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.41899999999999998,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.45400000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.45200000000000001,"15min":0.439},"min":{"1min":0.314,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.495},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.443,"15min":0.42999999999999999},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.41599999999999998,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.43099999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51400000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.34300000000000003,"15min":0.33800000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.311}]},{"osd":5,"last update":"Tue Sep 17 07:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35299999999999998,"15min":0.35199999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.378},{"interface":"front","average":{"1min":0.433,"5min":0.38900000000000001,"15min":0.38200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.47099999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320156,"kb_used_data":14484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94051684,"statfs":{"total":96636764160,"available":96308924416,"internally_reserved":0,"allocated":14831616,"data_stored":13552713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:52:52 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.27800000000000002},"min":{"1min":0.20999999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.29499999999999998,"15min":0.28699999999999998},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.29599999999999999,"15min":0.32100000000000001},"min":{"1min":0.25800000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.47599999999999998,"5min":0.49299999999999999,"15min":0.66400000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.30199999999999999,"15min":0.312},"min":{"1min":0.28599999999999998,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.35899999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:52:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.47399999999999998,"15min":0.45700000000000002},"min":{"1min":0.44800000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.46000000000000002,"15min":0.44600000000000001},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.41399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.439,"15min":0.46400000000000002},"min":{"1min":0.38900000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47099999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.44700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.41999999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640336,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361716,"kb_used_data":56044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94010124,"statfs":{"total":96636764160,"available":96266366976,"internally_reserved":0,"allocated":57389056,"data_stored":56086420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:52:48 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28699999999999998},"min":{"1min":0.21199999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.29499999999999998,"15min":0.29099999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.29699999999999999,"15min":0.33100000000000002},"min":{"1min":0.21299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.57699999999999996},"last":0.31},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.35499999999999998},"min":{"1min":0.28999999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.35399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.40400000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.47099999999999997,"15min":0.45700000000000002},"min":{"1min":0.42599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.53000000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.45400000000000001,"15min":0.47899999999999998},"min":{"1min":0.38600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.71999999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.46200000000000002,"15min":0.50700000000000001},"min":{"1min":0.42099999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.75,"5min":0.75,"15min":0.91300000000000003},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.47199999999999998,"15min":0.439},"min":{"1min":0.46700000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71799999999999997,"5min":0.746,"15min":0.746},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.47899999999999998,"15min":0.46500000000000002},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67900000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.63900000000000001}]}]},{"osd":0,"up_from":121,"seq":519691042847,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324896,"kb_used_data":26776,"kb_used_omap":4,"kb_used_meta":298107,"kb_avail":94046944,"statfs":{"total":96636764160,"available":96304070656,"internally_reserved":0,"allocated":27418624,"data_stored":26135621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":305262086},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.30099999999999999,"15min":0.29099999999999998},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54600000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.311},{"interface":"front","average":{"1min":0.375,"5min":0.29699999999999999,"15min":0.28399999999999997},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.378}]},{"osd":2,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.314,"15min":0.29999999999999999},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.31,"15min":0.29999999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.503,"15min":0.48599999999999999},"min":{"1min":0.45300000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.50800000000000001,"15min":0.48999999999999999},"min":{"1min":0.46300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56499999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.443,"15min":0.42599999999999999},"min":{"1min":0.38800000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.62},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.46000000000000002,"15min":0.44500000000000001},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.505}]},{"osd":5,"last update":"Tue Sep 17 07:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.45800000000000002,"15min":0.442},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.45300000000000001,"15min":0.437},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2798,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37777408,"data_stored":37749060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:53:11.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T07:53:11.899 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T07:53:11.899 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T07:53:11.899 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T07:53:11.900 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:11.934 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T07:53:11.934 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T07:53:11.934 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.008 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T07:53:12.008 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T07:53:12.009 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.083 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T07:53:12.084 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T07:53:12.084 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.121 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T07:53:12.121 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T07:53:12.122 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.191 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T07:53:12.191 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T07:53:12.191 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.261 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T07:53:12.261 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T07:53:12.261 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T07:53:12.261 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.329 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T07:53:12.329 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T07:53:12.330 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T07:53:12.330 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.341 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T07:53:12.342 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T07:53:12.342 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.393 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T07:53:12.394 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T07:53:12.394 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T07:53:12.407 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T07:53:12.407 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T07:53:12.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-09-17T07:53:12.468 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:53:12.468 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T07:53:12.617 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:53:12.618 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T07:53:12.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:53:12.741 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T07:53:12.824 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:53:12.825 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T07:53:12.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:53:12.903 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T07:53:12.992 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:53:12.992 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T07:53:13.051 INFO:tasks.ceph:Archiving mon data... 2024-09-17T07:53:13.052 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/data/mon.a.tgz 2024-09-17T07:53:13.052 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-17T07:53:13.068 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.3Ox5ctO4v0 2024-09-17T07:53:13.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3Ox5ctO4v0 2024-09-17T07:53:13.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.3Ox5ctO4v0 2024-09-17T07:53:13.329 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.3Ox5ctO4v0 is 435KB 2024-09-17T07:53:13.428 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.3Ox5ctO4v0 2024-09-17T07:53:13.443 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/data/mon.c.tgz 2024-09-17T07:53:13.444 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-17T07:53:13.499 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.hdxUJCBvFe 2024-09-17T07:53:13.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hdxUJCBvFe 2024-09-17T07:53:13.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.hdxUJCBvFe 2024-09-17T07:53:13.788 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.hdxUJCBvFe is 638KB 2024-09-17T07:53:13.888 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.hdxUJCBvFe 2024-09-17T07:53:13.904 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/data/mon.b.tgz 2024-09-17T07:53:13.904 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-09-17T07:53:13.922 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.lTqiXWyKZP 2024-09-17T07:53:13.922 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.lTqiXWyKZP 2024-09-17T07:53:14.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.lTqiXWyKZP 2024-09-17T07:53:14.222 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.lTqiXWyKZP is 638KB 2024-09-17T07:53:14.323 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.lTqiXWyKZP 2024-09-17T07:53:14.337 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T07:53:14.338 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T07:53:14.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T07:53:14.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T07:53:14.410 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-17T07:53:14.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-17T07:53:14.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-17T07:53:14.441 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-17T07:53:14.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-17T07:53:14.468 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-17T07:53:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-17T07:53:14.487 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T07:53:14.489 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi026/crash 2024-09-17T07:53:14.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T07:53:14.518 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi094/crash 2024-09-17T07:53:14.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T07:53:14.543 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi098/crash 2024-09-17T07:53:14.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T07:53:14.653 INFO:tasks.ceph:Compressing logs... 2024-09-17T07:53:14.653 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T07:53:14.656 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T07:53:14.658 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T07:53:14.683 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.49407.log 2024-09-17T07:53:14.683 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31517.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31353.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.49407.log: /var/log/ceph/ceph-client.admin.31517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31433.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31517.log.gz 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.31353.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41339.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31353.log.gz 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.31433.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.48568.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31433.log.gz 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.48568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31393.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.31393.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.47746.log 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31393.log.gz 2024-09-17T07:53:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40283.log 2024-09-17T07:53:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.47746.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.51154.log 2024-09-17T07:53:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.40283.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T07:53:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.51154.log: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T07:53:14.696 INFO:teuthology.orchestra.run.smithi094.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool.49407.log.gz 2024-09-17T07:53:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T07:53:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T07:53:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.log: 95.0% -- replaced with /var/log/ceph/objectstore_tool.41339.log.gz 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool.48568.log.gz 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr: 95.4% -- replaced with /var/log/ceph/objectstore_tool.47746.log.gz 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr: 95.2% -- replaced with /var/log/ceph/objectstore_tool.40283.log.gz 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31473.log 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.31473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31473.log.gz 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31313.log 2024-09-17T07:53:14.697 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-09-17T07:53:14.698 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mon.b.log: 95.1% -- replaced with /var/log/ceph/objectstore_tool.51154.log.gz 2024-09-17T07:53:14.698 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.31313.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.50232.log 2024-09-17T07:53:14.698 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31313.log.gz 2024-09-17T07:53:14.698 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool..log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T07:53:14.699 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/objectstore_tool.50232.log: /var/log/ceph/ceph-osd.3.log: 95.2% -- replaced with /var/log/ceph/objectstore_tool.50232.log.gz 2024-09-17T07:53:14.741 INFO:teuthology.orchestra.run.smithi094.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-09-17T07:53:14.807 INFO:teuthology.orchestra.run.smithi094.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T07:53:14.848 INFO:teuthology.orchestra.run.smithi094.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T07:53:14.893 INFO:teuthology.orchestra.run.smithi094.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T07:53:14.894 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28212.log 2024-09-17T07:53:14.894 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30397.log 2024-09-17T07:53:14.894 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28357.log 2024-09-17T07:53:14.894 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28212.log.gz 2024-09-17T07:53:14.894 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29767.log 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30397.log.gz 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30135.log 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28357.log.gz 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30585.log 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29767.log.gz 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29051.log 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30135.log.gz 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29588.log 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30585.log.gz 2024-09-17T07:53:14.895 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29991.log 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29051.log.gz 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28437.log 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29588.log.gz 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28227.log 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29991.log.gz 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27529.log 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28437.log.gz 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28908.log 0.0% 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28227.log.gz 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29243.log 2024-09-17T07:53:14.896 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27529.log.gz 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28359.log 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28908.log.gz 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30053.log 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29243.log.gz 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27669.log 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.28359.log.gz 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29264.log 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30053.log.gz 2024-09-17T07:53:14.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31336.log 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27669.log.gz 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28519.log 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29264.log.gz 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28603.log 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr: 51.3% -- replaced with /var/log/ceph/ceph-client.admin.31336.log.gz 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31357.log 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28519.log.gz 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28067.log 2024-09-17T07:53:14.898 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28603.log.gz 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28871.log 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31357.log.gz 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30186.log 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr: 31.8% -- replaced with /var/log/ceph/ceph-client.admin.28067.log.gz 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27409.log 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.28871.log.gz 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29232.log 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30186.log.gz 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28807.log 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27409.log.gz 2024-09-17T07:53:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30039.log 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29232.log.gz 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28807.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28807.log.gz 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29559.log 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28621.log 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30039.log.gz 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30408.log 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29559.log.gz 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29631.log 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28621.log.gz 2024-09-17T07:53:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29148.log 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.30408.log.gz 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28102.log 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr: 44.7% -- replaced with /var/log/ceph/ceph-client.admin.29631.log.gz 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27681.log 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.29148.log.gz 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27905.log 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28102.log.gz 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29196.log 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27681.log.gz 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27998.log 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.27905.log.gz 2024-09-17T07:53:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28020.log 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.29196.log.gz 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28448.log 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27998.log.gz 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28020.log.gz 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31211.log 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr: 52.6% -- replaced with /var/log/ceph/ceph-client.admin.28448.log.gz 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28770.log 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr: 45.5% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28555.log 2024-09-17T07:53:14.902 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31211.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28999.log 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.28770.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29435.log 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28555.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29969.log 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28999.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30223.log 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.29435.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29296.log 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29969.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30223.log.gz 2024-09-17T07:53:14.903 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27499.log 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28985.log 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.29296.log.gz 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27546.log 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr: 27.5% -- replaced with /var/log/ceph/ceph-client.admin.27499.log.gz 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28693.log 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28985.log.gz 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28704.log 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.27546.log.gz 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30904.log 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28693.log.gz 2024-09-17T07:53:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28419.log 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28704.log.gz 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29845.log 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30904.log.gz 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27739.log 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28419.log.gz 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29924.log 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29845.log.gz 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30764.log 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.27739.log.gz 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30342.log 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr: 44.3% -- replaced with /var/log/ceph/ceph-client.admin.29924.log.gz 2024-09-17T07:53:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27683.log 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr: 6.4% -- replaced with /var/log/ceph/ceph-client.admin.30764.log.gz 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27055.log 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30342.log.gz 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31374.log 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr: 18.0% -- replaced with /var/log/ceph/ceph-client.admin.27683.log.gz 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27903.log 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27055.log.gz 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27097.log 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31374.log.gz 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27903.log.gz 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28394.log 2024-09-17T07:53:14.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30120.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27097.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27013.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28394.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29728.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30120.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28860.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27013.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29728.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30117.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29360.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28860.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31160.log 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30117.log.gz 2024-09-17T07:53:14.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28108.log 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29360.log.gz 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29915.log 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31160.log.gz 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30285.log 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28108.log.gz 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28761.log 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr: 19.8% -- replaced with /var/log/ceph/ceph-client.admin.29915.log.gz 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30797.log 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30285.log.gz 2024-09-17T07:53:14.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30967.log 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.28761.log.gz 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30629.log 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30797.log.gz 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30183.log 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30967.log.gz 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28516.log 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30629.log.gz 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29983.log 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30183.log.gz 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29499.log 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28516.log.gz 2024-09-17T07:53:14.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31030.log 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29983.log.gz 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28182.log 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29499.log.gz 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27389.log 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31030.log.gz 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30043.log 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28182.log.gz 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28209.log 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27389.log.gz 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29014.log 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30043.log.gz 2024-09-17T07:53:14.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31047.log 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28209.log.gz 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28291.log 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29014.log.gz 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27771.log 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31047.log.gz 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27786.log 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28291.log.gz 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29426.log 2024-09-17T07:53:14.911 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27771.log.gz 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30697.log 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.27786.log.gz 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29205.log 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.29426.log.gz 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29157.log 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30697.log.gz 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28047.log 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.29205.log.gz 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29246.log 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr: 54.5% -- replaced with /var/log/ceph/ceph-client.admin.29157.log.gz 2024-09-17T07:53:14.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29490.log 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28047.log.gz 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29824.log 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29246.log.gz 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30064.log 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29490.log.gz 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30295.log 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29824.log.gz 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30883.log 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr: 34.0% -- replaced with /var/log/ceph/ceph-client.admin.30064.log.gz 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29369.log 2024-09-17T07:53:14.913 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.30295.log.gz 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30930.log 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30883.log.gz 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28264.log 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29369.log.gz 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27372.log 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30930.log.gz 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30225.log 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28264.log.gz 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28948.log 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27372.log.gz 2024-09-17T07:53:14.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28160.log 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.30225.log.gz 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28342.log 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28948.log.gz 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29590.log 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28160.log.gz 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29662.log 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28342.log.gz 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28796.log 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.29590.log.gz 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28911.log 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.29662.log.gz 2024-09-17T07:53:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31253.log 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28796.log.gz 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28911.log.gz 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29128.log 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31058.log 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr: 51.7% -- replaced with /var/log/ceph/ceph-client.admin.31253.log.gz 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29342.log 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29128.log.gz 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30662.log 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.31058.log.gz 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.29342.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30293.log 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29342.log.gz 2024-09-17T07:53:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29386.log 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.30662.log.gz 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28592.log 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30293.log.gz 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31316.log 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29386.log.gz 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29305.log 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28592.log.gz 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29472.log 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31316.log.gz 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29950.log 2024-09-17T07:53:14.917 INFO:teuthology.orchestra.run.smithi098.stderr: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.29305.log.gz 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27473.log 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29472.log.gz 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29950.log.gz 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28825.log 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27888.log 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27473.log.gz 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29572.log 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28825.log.gz 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31049.log 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27888.log.gz 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31127.log 2024-09-17T07:53:14.918 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29572.log.gz 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30346.log 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.31049.log.gz 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27795.log 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.31127.log.gz 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29194.log 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30346.log.gz 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30146.log 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr: 55.5% -- replaced with /var/log/ceph/ceph-client.admin.27795.log.gz 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29131.log 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29194.log.gz 2024-09-17T07:53:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29599.log 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.30146.log.gz 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29131.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29131.log.gz -5 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29626.log 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26971.log 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr: 61.1% -- replaced with /var/log/ceph/ceph-client.admin.29599.log.gz 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28921.log 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29626.log.gz 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31098.log 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26971.log.gz 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29577.log 2024-09-17T07:53:14.920 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28921.log.gz 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27820.log 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31098.log.gz 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28722.log 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29577.log.gz 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27825.log 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27820.log.gz 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28722.log.gz 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31178.log 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27825.log.gz 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27968.log 2024-09-17T07:53:14.921 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26929.log 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31178.log.gz 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28656.log 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27968.log.gz 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26929.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.27719.log 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26929.log.gz 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28528.log 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28656.log.gz 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27719.log.gz 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31175.log 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28528.log: /var/log/ceph/ceph-client.admin.29461.log 2024-09-17T07:53:14.922 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28528.log.gz 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28271.log 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31175.log.gz 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26886.log 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29461.log.gz 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29770.log 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28271.log.gz 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31391.log 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26886.log.gz 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28959.log 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29770.log.gz 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28439.log 2024-09-17T07:53:14.923 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31391.log.gz 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30517.log 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.28959.log.gz 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31118.log 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.28439.log.gz 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31116.log 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30517.log.gz 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29699.log 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.31118.log.gz 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30304.log 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31116.log.gz 2024-09-17T07:53:14.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28368.log 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29699.log.gz 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29016.log 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr: 45.8% -- replaced with /var/log/ceph/ceph-client.admin.30304.log.gz 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28076.log 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr: 53.3% -- replaced with /var/log/ceph/ceph-client.admin.28368.log.gz 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30436.log 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.29016.log.gz 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28489.log 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr: 56.2% -- replaced with /var/log/ceph/ceph-client.admin.28076.log.gz 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28050.log 2024-09-17T07:53:14.925 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30436.log.gz 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28474.log 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr: 41.2% -- replaced with /var/log/ceph/ceph-client.admin.28489.log.gz 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30330.log 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28050.log.gz 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31027.log 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28474.log.gz 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27768.log 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30330.log.gz 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31190.log 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31027.log.gz 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27429.log 2024-09-17T07:53:14.926 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27768.log.gz 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29524.log 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31190.log.gz 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27429.log.gzgzip 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28339.log 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30260.log 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.29524.log.gz 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31101.log 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28339.log.gz 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29183.log 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30260.log.gz 2024-09-17T07:53:14.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29896.log 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31101.log.gz 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29183.log.gz 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31001.log 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29896.log.gz 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30370.log 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31295.log 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31001.log.gz 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29913.log 2024-09-17T07:53:14.928 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30370.log.gz 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26833.log 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31295.log.gz 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29913.log.gz 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28732.log 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26833.log.gz 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30137.log 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28732.log.gz 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30350.log 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28630.log 2024-09-17T07:53:14.929 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.30137.log.gz 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27490.log 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30350.log.gz 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30196.log 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28630.log.gz 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30551.log 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr: 18.0% -- replaced with /var/log/ceph/ceph-client.admin.27490.log.gz 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29533.log 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30196.log.gz 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5/var/log/ceph/ceph-client.admin.30551.log: --verbose -- /var/log/ceph/ceph-client.admin.28566.log 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30551.log.gz 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28422.log 2024-09-17T07:53:14.930 INFO:teuthology.orchestra.run.smithi098.stderr: 27.0% -- replaced with /var/log/ceph/ceph-client.admin.29533.log.gz 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28238.log 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.28566.log.gz 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28422.log.gz 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30271.log 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30932.log 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr: 52.0% -- replaced with /var/log/ceph/ceph-client.admin.28238.log.gz 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30271.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.30275.log -- replaced with /var/log/ceph/ceph-client.admin.30271.log.gz 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27946.log 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.30932.log.gz 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28667.log 2024-09-17T07:53:14.931 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30275.log.gz 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28996.log 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27946.log.gz 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27555.log 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.28667.log.gz 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28065.log 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28996.log.gz 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31274.log 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr: 4.8% -- replaced with /var/log/ceph/ceph-client.admin.27555.log.gz 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28950.log 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28065.log.gz 2024-09-17T07:53:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27139.log 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31274.log.gz 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29062.log 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.28950.log.gz 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29279.log 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27139.log.gz 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27590.log 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29062.log.gz 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29522.log 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29279.log.gz 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30172.log 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27590.log: 27.5% -- replaced with /var/log/ceph/ceph-client.admin.27590.log.gz 2024-09-17T07:53:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29261.log 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29522.log.gz 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30090.log 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30172.log.gz 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28557.log 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29261.log.gz 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28130.log 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30090.log.gz 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27453.log 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr: 18.0% -- replaced with /var/log/ceph/ceph-client.admin.28557.log.gz 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28862.log 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28130.log.gz 2024-09-17T07:53:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28312.log 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27453.log.gz 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27672.log 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr: 18.0% -- replaced with /var/log/ceph/ceph-client.admin.28862.log.gz 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27858.log 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28312.log.gz 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29065.log 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27672.log.gz 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr: 53.0% -- replaced with /var/log/ceph/ceph-client.admin.27858.log.gz 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29065.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.29065.log.gz 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31232.log 2024-09-17T07:53:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29091.log 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr: 21.3% -- replaced with /var/log/ceph/ceph-client.admin.31232.log.gz 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30399.log 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr: 19.8% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29146.log 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.29091.log.gz 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31084.log 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.30399.log.gz 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27914.log 2024-09-17T07:53:14.936 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29146.log.gz 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31084.log.gz 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27940.log 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27692.log 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr: 51.6% -- replaced with /var/log/ceph/ceph-client.admin.27914.log.gz 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30234.log 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27940.log.gz 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27885.log 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr: 55.8% -- replaced with /var/log/ceph/ceph-client.admin.27692.log.gz 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27730.log 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr: 43.6% -- replaced with /var/log/ceph/ceph-client.admin.30234.log.gz 2024-09-17T07:53:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27581.log 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27885.log.gz 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30941.log 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr: 18.0% -- replaced with /var/log/ceph/ceph-client.admin.27730.log.gz 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30972.log 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.27581.log.gz 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28897.log 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr: 55.9% -- replaced with /var/log/ceph/ceph-client.admin.30941.log.gz 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28759.log 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30972.log.gz 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30831.log 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28897.log.gz 2024-09-17T07:53:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28759.log.gz 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28658.log 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29409.log 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30831.log.gz 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30874.log 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.28658.log.gz 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29082.log 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29409.log.gz 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30484.log 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30874.log.gz 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29331.log 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.29082.log.gz 2024-09-17T07:53:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30028.log 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30484.log.gz 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28229.log 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29331.log.gz 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30870.log 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30028.log.gz 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29025.log 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.28229.log.gz 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29117.log 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27667.log 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.29025.log.gz 2024-09-17T07:53:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29973.log 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29117.log.gz 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30730.log 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27667.log.gz 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29973.log.gz 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30730.log: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30730.log.gz 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m0.087s 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m0.088s 2024-09-17T07:53:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.117s 2024-09-17T07:53:14.943 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T07:53:15.037 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82706.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47350.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82706.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.82706.log.gz 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.82614.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.47350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81361.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47350.log.gz 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82614.log.gzgzip 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63606.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72613.log 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81361.log.gz 2024-09-17T07:53:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63606.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63606.log.gz 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.61216.log 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.72613.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82202.log 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72613.log.gz 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61216.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.61216.log.gz -- 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.57114.log 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81976.log 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82202.log.gz 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.57114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57114.log.gz 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.43399.log 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73311.log 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81976.log.gz 2024-09-17T07:53:15.039 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.43399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82494.log 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36824.log 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73311.log.gz 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78152.log 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82494.log.gz 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36824.log.gz 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47092.log 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59946.log 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78152.log.gz 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.47092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47092.log.gz 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63517.log 2024-09-17T07:53:15.040 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68522.log 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59946.log.gz 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63517.log.gz 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66737.log 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68522.log.gz 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66737.log.gz 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83420.log 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75825.log 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-09-17T07:53:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83420.log.gz 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67093.log 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36043.log 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75825.log.gz 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60108.log 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67093.log.gz 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool.43399.log.gz 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36043.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41449.log 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36043.log.gz 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36547.log 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60108.log.gz 2024-09-17T07:53:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.41449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35830.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36547.log.gz 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79877.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65394.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35830.log.gz 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79877.log.gz 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76860.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66566.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65394.log.gz 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76860.log.gz 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68735.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83714.log 2024-09-17T07:53:15.043 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66566.log.gz 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68735.log.gz 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81062.log 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63355.log 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83714.log.gz 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81062.log.gz 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38555.log 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58678.log 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63355.log.gz 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38555.log.gz 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64994.log 2024-09-17T07:53:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.58678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38292.log 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58678.log.gz 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.64994.log: 95.2% -- replaced with /var/log/ceph/objectstore_tool.41449.log.gz 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64994.log.gz 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73014.log 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39044.log 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38292.log.gz 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73014.log.gz 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39044.log.gz 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T07:53:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82322.log 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72633.log 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82322.log.gz 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35821.log 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31654.log 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72633.log.gz 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35821.log.gz 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69760.log 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31654.log.gz 2024-09-17T07:53:15.046 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80834.log 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70113.log 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69760.log.gz 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80834.log.gz 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83126.log 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.70113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65557.log 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70113.log.gz 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83126.log.gz 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71307.log 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65557.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.50079.log 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65557.log.gz 2024-09-17T07:53:15.047 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71307.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.71307.log.gz 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72327.log 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.50079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36396.log 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72327.log.gz 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48279.log 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63851.log 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36396.log.gz 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48279.log.gz 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39770.log 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77220.log 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63851.log.gz 2024-09-17T07:53:15.048 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48920.log 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39770.log.gz 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77220.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.77220.log.gz 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.76379.log 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27253.log 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48920.log.gz 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76379.log.gz 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52437.log 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72994.log 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.tmp-client.admin.27253.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27253.log.gz 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.52437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52437.log.gz 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-09-17T07:53:15.049 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80063.log 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72994.log.gz 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69067.log 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80063.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.80063.log.gz 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.78600.log 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83588.log 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69067.log.gz 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78600.log: 95.3% -- replaced with /var/log/ceph/objectstore_tool.50079.log.gz 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78600.log.gz 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72912.log 2024-09-17T07:53:15.050 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59192.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83588.log.gz 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37649.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72912.log.gz 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51937.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59192.log.gz 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37649.log.gz 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36910.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.51937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51003.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51937.log.gz 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36910.log.gz 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54540.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.51003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58900.log 2024-09-17T07:53:15.051 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51003.log.gz 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54540.log.gz 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36296.log 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.58900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77219.log 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58900.log.gz 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36296.log.gz 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82320.log 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31966.log 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77219.log.gz 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55377.log 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82320.log.gz 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36013.log 2024-09-17T07:53:15.052 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31966.log.gz 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55377.log.gz 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.49145.log 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73803.log 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36013.log.gz 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.49145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62353.log 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77968.log 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73803.log.gz 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54349.log 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62353.log.gz 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61811.log 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77968.log.gz 2024-09-17T07:53:15.053 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36668.log 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54349.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59212.log 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61811.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37964.log 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36668.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83252.log 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59212.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70169.log 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37964.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55478.log 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83252.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.70169.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70169.log.gz 2024-09-17T07:53:15.054 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.74599.log 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55478.log.gz 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82362.log 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36628.log 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74599.log.gz 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82362.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82362.log.gz 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72768.log 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr: 95.4% -- replaced with /var/log/ceph/objectstore_tool.49145.log.gz 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36628.log.gz 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58589.log 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82262.log 2024-09-17T07:53:15.055 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72768.log.gz 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.58589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35832.log 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58589.log.gz 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82262.log.gz 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36086.log 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79365.log 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35832.log.gz 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36086.log.gz 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64474.log 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83504.log 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79365.log.gz 2024-09-17T07:53:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.64474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64474.log.gz 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72253.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72028.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83504.log.gz 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72253.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.72253.log.gz 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48450.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72028.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.44251.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72028.log.gz 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48450.log.gz 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83378.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.44251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54935.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62081.log 2024-09-17T07:53:15.057 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83378.log.gz 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54935.log.gz 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60814.log 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52829.log 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62081.log.gz 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60814.log.gz 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44735.log 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80199.log 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.52829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52829.log.gz 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.44735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44735.log.gz 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45195.log 2024-09-17T07:53:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57947.log 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80199.log.gz 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.45195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41453.log 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45195.log.gz 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.57947.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.57947.log.gz 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62522.log 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41453.log: /var/log/ceph/ceph-client.admin.83042.log 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41453.log.gz 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62522.log.gz 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75652.log 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83462.log 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83042.log.gz 2024-09-17T07:53:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81403.log 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75652.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83462.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83084.log 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr: 95.1% -- replaced with /var/log/ceph/objectstore_tool.44251.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81403.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77895.log 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57514.log 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83084.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77895.log.gz 2024-09-17T07:53:15.060 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53775.log 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.57514.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57514.log.gz 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53775.log.gz 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82832.log 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35648.log 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36782.log 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82832.log.gz 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35648.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35648.log.gz 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72469.log 2024-09-17T07:53:15.061 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57615.log 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36782.log.gz 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64725.log 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72469.log.gz 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.57615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57615.log.gz 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77099.log 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.64725.log: /var/log/ceph/ceph-client.admin.37176.log 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64725.log.gz 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81797.log 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77099.log.gz 2024-09-17T07:53:15.062 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37176.log: 93.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71117.log 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37176.log.gz 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81797.log.gz 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71117.log.gz 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80615.log 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63932.log 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-09-17T07:53:15.063 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80615.log.gz 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77948.log 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53967.log 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63932.log.gz 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77948.log.gz 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81020.log 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62914.log 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53967.log.gz 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81020.log.gz 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78987.log 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36508.log 2024-09-17T07:53:15.064 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62914.log.gz 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78987.log.gz 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71368.log 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61105.log 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36508.log.gz 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71368.log.gz 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65686.log 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62190.log 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61105.log.gz 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65686.log.gz 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75191.log 2024-09-17T07:53:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82050.log 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62190.log.gz 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75191.log.gz 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55105.log 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36044.log 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82050.log.gz 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55105.log.gz 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41414.log 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79707.log 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36044.log.gz 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41414.log.gz 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37056.log 2024-09-17T07:53:15.066 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81319.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79707.log.gz 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37056.log.gz 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80936.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56905.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81319.log.gz 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80936.log.gz 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48758.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66525.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56905.log.gz 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48758.log.gz 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61085.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49579.log 2024-09-17T07:53:15.067 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66525.log.gz 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39998.log 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61085.log.gz 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.49579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71953.log 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49579.log.gz 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67029.log 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60463.log 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71953.log.gz 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67029.log.gz 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54723.log 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71580.log 2024-09-17T07:53:15.068 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60463.log.gz 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54723.log.gz 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83336.log 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35820.log 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71580.log.gz 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36256.log 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83336.log.gz 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66040.log 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35820.log.gz 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41164.log 2024-09-17T07:53:15.069 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66040.log.gz 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83630.log 0.0% 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45707.log 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41164.log.gz 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83630.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.82574.log -- replaced with /var/log/ceph/ceph-client.admin.83630.log.gz 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.45707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52518.log 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45707.log.gz 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78875.log 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82574.log.gz 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.52518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81792.log 2024-09-17T07:53:15.070 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52518.log.gz 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78875.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.78875.log.gz 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49091.log 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81796.log 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81792.log.gz 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.49091.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49091.log.gz 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.36588.log 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66778.log 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81796.log.gz 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36588.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz -- 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.69321.log 2024-09-17T07:53:15.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83294.log 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66778.log.gz 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67705.log 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69321.log.gz 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38669.log 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83294.log.gz 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77620.log 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67705.log.gz 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82008.log 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38669.log.gz 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75926.log 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77620.log.gz 2024-09-17T07:53:15.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36236.log 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82008.log.gz 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36488.log 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75926.log.gz 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68836.log 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gz 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36488.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36488.log.gz 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55861.log 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68836.log.gz 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45972.log 2024-09-17T07:53:15.073 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55861.log.gz 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36952.log 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59615.log 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.45972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45972.log.gz 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36952.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.51051.log 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36952.log.gz 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81789.log 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59615.log.gz 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69157.log 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.51051.log: /var/log/ceph/ceph-client.admin.81789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48196.log 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81789.log.gz 2024-09-17T07:53:15.074 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52871.log 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69157.log.gz 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79660.log 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48196.log.gz 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.52871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82790.log 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52871.log.gz 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79660.log.gz 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81975.log 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81193.log 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82790.log.gz 2024-09-17T07:53:15.075 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81975.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81975.log.gz 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.82516.log 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81193.log.gz 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76656.log 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78386.log 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82516.log.gz 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.76656.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81571.log 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76656.log.gz 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56383.log 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78386.log.gz 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81571.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81571.log.gz 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.37179.log 2024-09-17T07:53:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56383.log.gz 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83546.log 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37179.log.gz 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80514.log 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82493.log 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83546.log.gz 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52189.log 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80514.log.gz 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83000.log 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82493.log.gz 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr: 95.2% -- replaced with /var/log/ceph/objectstore_tool.51051.log.gz 2024-09-17T07:53:15.077 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.52189.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64583.log 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52189.log.gz 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83000.log.gz 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54219.log 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77324.log 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.64583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64583.log.gz 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54219.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54219.log.gz 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83796.log 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77324.log.gz 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54470.log 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83796.log.gz 2024-09-17T07:53:15.078 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83672.log 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81277.log 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54470.log.gz 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83672.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83672.log.gz 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.39812.log 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81277.log.gz 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53041.log 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81613.log 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36015.log 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53041.log.gz 2024-09-17T07:53:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81613.log.gz 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58338.log 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36015.log.gz 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38005.log 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.58338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60564.log 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58338.log.gz 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67766.log 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38005.log.gz 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58278.log 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60564.log.gz 2024-09-17T07:53:15.080 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81655.log 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67766.log: /var/log/ceph/ceph-client.admin.58278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76975.log 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58278.log.gz 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67766.log.gz 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66041.log 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81655.log.gz 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82009.log 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76975.log.gz 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66041.log: /var/log/ceph/ceph-client.admin.82009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66041.log.gz 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69471.log 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82009.log.gz 2024-09-17T07:53:15.081 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82916.log 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36740.log 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69471.log.gz 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54077.log 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82916.log.gz 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36740.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36740.log.gz 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37177.log 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54077.log.gz 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64183.log 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37177.log.gz 2024-09-17T07:53:15.082 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77384.log 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.64183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64183.log.gz 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48542.log 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77384.log.gz 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73701.log 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48542.log.gz 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67161.log 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73701.log.gz 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56654.log 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67161.log.gz 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55649.log 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56654.log.gz 2024-09-17T07:53:15.083 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60901.log 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55649.log.gz 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74546.log 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60901.log.gz 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74546.log.gz 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74862.log 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74862.log.gz 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71639.log 2024-09-17T07:53:15.084 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79426.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71639.log.gz 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75550.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79426.log.gz 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71853.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75550.log.gz 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68107.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.71853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71853.log.gz 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35831.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68107.log.gz 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83756.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35831.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57778.log 2024-09-17T07:53:15.085 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35831.log.gz 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77640.log 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83756.log.gz 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53151.log 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.57778.log: /var/log/ceph/ceph-client.admin.77640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76100.log 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77640.log.gz 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57778.log.gz 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53151.log.gz 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76100.log.gz 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75335.log 2024-09-17T07:53:15.086 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35690.log 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75335.log.gz 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82748.log 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35690.log.gz 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50943.log 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82748.log.gz 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78305.log 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.50943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50943.log.gz 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80710.log 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78305.log.gz 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59284.log 2024-09-17T07:53:15.087 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80710.log.gz 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35515.log 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59284.log.gz 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68037.log 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35515.log.gz 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53504.log 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68037.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62813.log 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68037.log.gz 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53504.log.gz 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79110.log 2024-09-17T07:53:15.088 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62813.log: gzip/var/log/ceph/ceph-client.admin.40764.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.78673.log 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0%/var/log/ceph/ceph-client.admin.79110.log: -- replaced with /var/log/ceph/ceph-client.admin.62813.log.gz 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79110.log.gz 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.78673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78673.log.gz 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76901.log 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73384.log 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75292.log 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76901.log.gz 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73384.log.gz 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74384.log 2024-09-17T07:53:15.089 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80125.log 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75292.log.gz 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74384.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59068.log 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55699.log 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80125.log: 0.0%/var/log/ceph/ceph-client.admin.59068.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59068.log.gz 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.79815.log 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80125.log.gz 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55699.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73996.log 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55699.log.gz 2024-09-17T07:53:15.090 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53423.log 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79815.log.gz 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68460.log 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73996.log: /var/log/ceph/ceph-client.admin.53423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75056.log 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53423.log.gz 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73996.log.gz 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68460.log.gz 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48707.log 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75056.log.gz 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72429.log 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48707.log.gz 2024-09-17T07:53:15.091 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53273.log 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72429.log.gz 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80792.log 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53273.log.gz 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82051.log 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80792.log.gz 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61711.log 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82051.log.gz 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62935.log 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61711.log.gz 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83168.log 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62935.log.gz 2024-09-17T07:53:15.092 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72830.log 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83168.log.gz 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81109.log 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72830.log.gz 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79939.log 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81109.log.gz 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35432.log 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79939.log.gz 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74721.log 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35432.log.gz 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76501.log 2024-09-17T07:53:15.093 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74721.log.gz 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59817.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76501.log.gz 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56743.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59817.log.gz 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60209.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56743.log.gz 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.42537.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60209.log.gz 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56031.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool.42537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81487.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56031.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37456.log 2024-09-17T07:53:15.094 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56031.log.gz 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35829.log 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81487.log.gz 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37456.log.gz 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68585.log 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35829.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52007.log 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35829.log.gz 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77057.log 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68585.log.gz 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.52007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52007.log.gz 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68269.log 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82454.log 2024-09-17T07:53:15.095 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.77057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77057.log.gz 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68269.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69553.log 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40622.log 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82454.log.gz 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69553.log.gz 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76249.log 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67413.log 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40622.log.gz 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76249.log: 0.0% 95.2% -- replaced with /var/log/ceph/objectstore_tool.42537.log.gz 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76249.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82222.log 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.096 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67322.log 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67413.log: /var/log/ceph/ceph-client.admin.82222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82222.log.gz 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67413.log.gz 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67322.log.gz 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76676.log 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68379.log 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39727.log: gzip/var/log/ceph/ceph-client.admin.76676.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76676.log.gz 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.81529.log 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.68379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48066.log 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68379.log.gz 2024-09-17T07:53:15.097 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39412.log 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81529.log.gz 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48066.log: /var/log/ceph/ceph-client.admin.39412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36353.log 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48066.log.gz 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39412.log.gz 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76541.log 2024-09-17T07:53:15.098 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31883.log 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36353.log: /var/log/ceph/ceph-client.admin.76541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72008.log 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36353.log.gz 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76541.log.gz 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31883.log.gz 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42479.log 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61318.log 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72008.log: /var/log/ceph/ceph-client.admin.42479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69836.log 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42479.log.gz 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72008.log.gz 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61318.log.gz 2024-09-17T07:53:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35606.log 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69836.log.gz 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62683.log 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35606.log.gz 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81151.log 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.62683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62683.log.gz 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56323.log 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81151.log.gz 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80341.log 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56323.log.gz 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39936.log 2024-09-17T07:53:15.100 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80341.log.gz 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48491.log 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39936.log.gz 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81795.log 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.48491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48491.log.gz 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38454.log 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81795.log.gz 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82874.log 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38454.log.gz 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82874.log.gz 2024-09-17T07:53:15.101 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37711.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/objectstore_tool..log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57425.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37711.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57235.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59566.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.57235.log: /var/log/ceph/ceph-client.admin.57425.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57235.log.gz 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57425.log.gz 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63226.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.59566.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66253.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59566.log.gz 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36868.log 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63226.log: /var/log/ceph/ceph-client.admin.66253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63226.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66253.log.gz 2024-09-17T07:53:15.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81794.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36868.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63688.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36868.log.gz 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76141.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81794.log: /var/log/ceph/ceph-client.admin.63688.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81794.log.gz 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63688.log.gz 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61359.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81445.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76141.log.gz 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61359.log.gzgzip 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54773.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82958.log 2024-09-17T07:53:15.103 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.54773.log: /var/log/ceph/ceph-client.admin.81445.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54773.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81445.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74056.log 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83210.log 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.82958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82958.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74056.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55227.log 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.83210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83210.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74271.log 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59129.log 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.55227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55227.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.74271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74271.log.gz 2024-09-17T07:53:15.104 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72212.log 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38239.log 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.72212.log: /var/log/ceph/ceph-client.admin.59129.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72212.log.gz 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59129.log.gz 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40335.log 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65055.log 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40335.log: /var/log/ceph/ceph-client.admin.38239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40335.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38239.log.gz 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73425.log 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53755.log 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65055.log: /var/log/ceph/ceph-client.admin.73425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65055.log.gz 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73425.log.gz 2024-09-17T07:53:15.105 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65819.log 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76162.log 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.53755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53755.log.gz 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65819.log.gz 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61522.log 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52667.log 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.76162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76162.log.gz 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.61522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61522.log.gz 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81235.log 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65284.log 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.52667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52667.log.gz 2024-09-17T07:53:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.81235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81235.log.gz 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63770.log 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36356.log 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65284.log.gz 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.63770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63770.log.gzgzip 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40847.log 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60170.log 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40847.log: /var/log/ceph/ceph-client.admin.36356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40847.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36356.log.gz 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75055.log 2024-09-17T07:53:15.107 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60170.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60170.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80001.log 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75055.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58049.log 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80001.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36994.log 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.58049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58049.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73097.log 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36994.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60321.log 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73097.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65768.log 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.60321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60321.log.gz 2024-09-17T07:53:15.108 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66363.log 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.65768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65768.log.gz 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79223.log 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.66363.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66363.log.gz 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44195.log 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.79223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79223.log.gz 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73528.log 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.44195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44195.log.gz 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56152.log 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.73528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73528.log.gz 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80978.log 2024-09-17T07:53:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.56152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56152.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75436.log 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.80978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80978.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40061.log 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.75436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75436.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64232.log 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40061.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69117.log 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.64232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64232.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67535.log 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.69117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69117.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.67535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67535.log.gz 2024-09-17T07:53:15.110 INFO:teuthology.orchestra.run.smithi026.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T07:53:15.111 INFO:teuthology.orchestra.run.smithi026.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T07:53:15.111 INFO:teuthology.orchestra.run.smithi026.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T07:53:15.112 INFO:teuthology.orchestra.run.smithi094.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T07:53:15.112 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-17T07:53:15.112 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m0.409s 2024-09-17T07:53:15.112 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m1.155s 2024-09-17T07:53:15.113 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m0.069s 2024-09-17T07:53:15.144 INFO:teuthology.orchestra.run.smithi026.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T07:53:15.447 INFO:teuthology.orchestra.run.smithi026.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T07:53:15.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-17T07:53:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m0.778s 2024-09-17T07:53:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m1.956s 2024-09-17T07:53:15.449 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.320s 2024-09-17T07:53:15.449 INFO:tasks.ceph:Archiving logs... 2024-09-17T07:53:15.449 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi026/log 2024-09-17T07:53:15.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:53:16.048 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi094/log 2024-09-17T07:53:16.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:53:16.245 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi098/log 2024-09-17T07:53:16.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:53:16.453 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T07:53:16.479 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T07:53:16.480 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T07:53:16.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T07:53:16.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T07:53:16.582 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T07:53:16.591 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T07:53:16.591 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:53:16.594 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:53:16.596 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:53:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:53:16.610 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:53:16.613 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:53:16.614 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:53:16.614 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-09-17T07:53:16.614 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +711us[ +712us] +/- 46ms 2024-09-17T07:53:16.614 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -4142us[-4142us] +/- 74ms 2024-09-17T07:53:16.614 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +2998us[+2999us] +/- 76ms 2024-09-17T07:53:16.614 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:53:16.616 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:53:16.616 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-09-17T07:53:16.616 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +746us[ +742us] +/- 46ms 2024-09-17T07:53:16.616 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -4101us[-4101us] +/- 74ms 2024-09-17T07:53:16.616 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +3033us[+3033us] +/- 76ms 2024-09-17T07:53:16.616 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:53:16.619 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:53:16.619 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-17T07:53:16.619 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 33 +715us[ +769us] +/- 46ms 2024-09-17T07:53:16.619 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 35 -4128us[-4074us] +/- 74ms 2024-09-17T07:53:16.619 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +2992us[+3046us] +/- 76ms 2024-09-17T07:53:16.619 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:53:16.619 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T07:53:16.686 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T07:53:16.686 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T07:53:16.721 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:53:16.824 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:53:16.912 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:53:16.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:53:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:53:17.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:53:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:53:17.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:53:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:53:17.074 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T07:53:17.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T07:53:17.133 INFO:teuthology.task.internal:Duration was 788.052182 seconds 2024-09-17T07:53:17.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T07:53:17.147 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T07:53:17.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:53:17.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:53:17.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:53:17.190 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:53:17.191 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:53:17.193 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:53:17.603 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T07:53:17.604 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-09-17T07:53:17.604 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:53:17.629 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-09-17T07:53:17.629 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:53:17.666 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-09-17T07:53:17.666 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:53:17.728 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T07:53:17.728 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:53:17.731 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:53:17.733 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:53:17.761 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T07:53:17.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:53:17.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:53:17.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:53:17.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T07:53:17.971 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T07:53:17.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:53:17.997 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:53:18.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:53:18.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T07:53:18.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:53:18.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:53:18.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:53:18.099 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-09-17T07:53:18.100 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-09-17T07:53:18.110 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-09-17T07:53:18.134 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:53:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:53:18.176 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:53:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:53:18.191 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:53:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:53:18.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T07:53:18.214 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T07:53:18.215 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi026 2024-09-17T07:53:18.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:53:18.252 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi094 2024-09-17T07:53:18.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:53:18.286 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-17_07:28:46-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7908725/remote/smithi098 2024-09-17T07:53:18.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:53:18.320 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T07:53:18.320 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:53:18.322 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:53:18.325 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:53:18.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T07:53:18.389 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T07:53:18.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T07:53:18.412 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T07:53:18.413 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:53:18.416 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:53:18.419 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:53:18.429 INFO:teuthology.orchestra.run.smithi026.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:53 /home/ubuntu/cephtest 2024-09-17T07:53:18.431 INFO:teuthology.orchestra.run.smithi094.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:53 /home/ubuntu/cephtest 2024-09-17T07:53:18.434 INFO:teuthology.orchestra.run.smithi098.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:53 /home/ubuntu/cephtest 2024-09-17T07:53:18.435 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T07:53:18.478 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T07:53:18.598 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_latest} thrashers/cache thrashosds-health workloads/rbd_nbd} duration: 788.0521821975708 flavor: default owner: scheduled_sangadi@teuthology success: true 2024-09-17T07:53:18.598 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T07:53:18.701 INFO:teuthology.run:pass